Acronyms 


ACAT  Acquisition  Category 
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INSPECTOR  GENERAL 
DEPAFtTMENT  OF  DEFENSE 
400  ARMY  NAVY  DRIVE 
ARLINGTON.  VIRGINIA  22202 


June  30,  1993 

MEMORANDUM  FOR  UNDER  SECRETARY  OF  DEFENSE  (ACQUISITION  AND 

TECHNOLOGY) 

ASSISTANT  SECRETARY  OF  THE  AIR  FORCE 

(FINANCIAL  MANAGEMENT  AND  COMPTROLLER) 

SUBJECT:  Quick-Reaction  Report  on  the  Acquisition  of  the  F-15  Downsized  Tester 
(Report  No.  93-138) 


We  are  providing  this  final  quick-reaction  report  for  your  information  and  use. 

It  addresses  the  planned  acquisition  of  the  F-15  Downsized  Tester  to  be  used  to  test  a 
portion  of  the  electronic  equipment  on  the  F-15  mrcraft  at  the  intermediate  maintenance 
level.  Comments  from  the  Principal  Deputy  Assistant  Secretary  of  Defense 
(Production  and  Logistics)  and  the  Deputy  Assistant  Secretary  of  the  Air  Force 
(Acquisition)  were  considered  in  preparing  this  final  report. 

Recommendations  are  subject  to  resolution  in  accordance  with  DoD 
Directive  7650.3  in  the  event  of  nonconcurrence  or  failure  to  comment.  Therefore, 
you  must  provide  final  comments  on  the  unresolved  recommendations  by 
July  15,  1993.  We  also  ask  that  your  comments  indicate  concurrence  or 
nonconcurrence  with  the  material  internal  control  weaknesses  highlighted  in  Part  I. 
Potential  monetary  benefits  were  undeterminable. 

The  cooperation  and  courtesies  extended  to  the  audit  staff  are  appreciated. 

Please  contact  Mr.  Christian  Hendricks,  Program  Director,  at  (703)  692-3414  (DSN 
222-3414)  or  Mr.  Tilghman  Schraden,  Project  Manager,  at  (703)  692-3413  (DSN  222- 
3413),  if  you  have  any  questions  concerning  this  final  quick-reaction  report.  Copies  of 
the  final  quick-reaction  report  will  be  distributed  to  the  activities  listed  in  Appendix  E. 


Robert  J.  Lieberman 
Assistant  Inspector  General 
for  Auditing 


Office  of  the  Inspector  General,  DoD 


June  30,  1993 


Report  No.  93-138 
Project  No.  3LB-5014 

ACQXnSITION  OF  THE  F-15  DOWNSIZED  TESTER 
EXECUTIVE  SUMMARY 


Introduction.  The  Air  Force  is  procuring  an  F-15  Downsized  Tester  (the  Tester)  to 
replace  existing  automatic  test  equipment  for  the  F-15  aircraft.  The  Tester  will  be  a 
portable  automatic  test  equipment  system  for  testing  a  portion  of  the  electronic 
equipment  on  the  F-15  aircraft  at  the  intermediate  maintenance  level.  The  Tester  will 
initially  replace  the  F-15  Avionics  Intermediate  Shop  test  equipment  that  has  been  in 
operation  since  1974.  Later  the  Tester  will  replace  the  F-15E  Mobile  Electronic  Test 
Set  used  on  the  F-15E  model  aircraft  since  1986. 

The  audit  was  requested  by  the  Director,  Weapon  Support  Irnprovement  Group,  Office 
of  the  Assistant  Secretaiw  of  Defense  (Production  and  Logistics).  The  Air  Force  plans 
to  award  a  contract  for  $167  million  for  55  units  of  automatic  test  equipment  over  the 
6-year  Future  Years  Defense  Program  (FY  1992  through  FY  1997). 

Objective.  The  primary  objective  of  the  audit  was  to  evaluate  Air  Force's  acquisition 
of  unique  test  equipment  to  replace  existing  F-15  avionics  test  equipment. 

Audit  Results.  The  Air  Force  was  planning  to  develop  and  procure  unique  automatic 
test  equipment  that  is  not  cost-effective.  As  a  result,  the  Air  Force  could  spend 
$167  million  for  automatic  test  equipment  with  capabilities  limited  to  testing  portions  of 
the  F-15  avionics  systems. 

Internal  Controls.  Internal  controls  were  not  effective  to  ensure  that  automatic  test 
equipment  being  acquired  for  the  F-15  aircraft  was  properly  justified  and  cost-effective. 
The  internal  control  weaknesses  are  discussed  in  detail  in  the  report. 

Potential  Benefits  of  Audit.  Undeterminable  savings  will  be  achieved  through 
standardization  of  equipment  and  support.  Appendix  C  summarizes  the  potential 
benefits  resulting  from  audit. 

Summary  of  Recommendations.  We  recommended  that  the  Air  Force  discontinue  its 
full  and  open  competitive  acquisition  of  the  Tester.  We  also  recommended  that 
thorough  analyses  of  costs  and  technical  requirements  be  properly  prepared  and 
approved  for  standardized  automatic  test  equipment.  Additionally,  we  recommended 
that  the  Under  Secretary  of  Defense  (Acquisition  and  Technology)  and  the  Air  Force 
implement  policies  and  procedures  for  standardizing  automatic  test  equipment. 

Management  Comments.  The  Air  Force  nonconcurred  with  the  recommendations  to 
discontinue  the  Tester  acquisition  and  to  prepare  thorough  analyses  of  costs  and 
technical  requirements,  stating  that  discontinuing  the  full  and  open  competition  would 
result  in  monetary  losses,  that  the  operational  requirements  document  was  approved 
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by  the  Air  Force  Vice  Chief  of  Staff  on  May  11,  1993,  and  that  a  cost  and  operational 
effectiveness  analysis  was  not  required.  The  Principal  Deputy  Assistant  Secretary  of 
Defense  (Production  and  Logistics)  and  the  Air  Force  concurred  with  recommendations 
to  implement  policies  and  procedures  for  standardizing  automatic  test  equipment.  The 
Principal  Deputy  also  advised  that  corrective  action  by  the  Under  Secretary  of  Defense 
(Acquisition  and  Technology)  would  be  more  appropriate.  The  complete  text  of 
management  comments  is  in  Part  IV  of  this  report. 

Audit  Response.  Benefits  from  developing  and  acquiring  standardize  automatic  test 
equipment  should  exceed  any  monetary  losses  incurred  by  the  Air  Force.  The 
operation^  requirements  document  and  cost  benefit  analysp  that  the  Air  Force 
prepared  were  based  on  unsupported  and  incomplete  information.  Consequentty,  we 
believe  more  thorough  analyses  are  necessary  to  ensure  that  the  Air  Force  acquires  the 
most  cost-effective  standardized  automatic  test  equipment.  Audit  responses  to  specific 
Air  Force  comments  are  in  Appendix  B.  We  request  that  the  Under  S^retary  of 
Defense  (Acquisition  and  Technology)  respond  to  the  revised  recommendation  and  the 
Assistant  Secretary  of  the  Air  Force  (Acquisition)  provide  comments  on  the  final  report 
by  July  15,  1993. 
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This  report  was  prepared  by  the  Logistics  Support  Directorate,  Office  of  the  Inspector 
General  for  Auditing,  DoD.  Copies  of  the  report  can  be  obtained  from  the  Secondary 
Reports  Distribution  Unit,  Audit  Planning  and  Technical  Support  Directorate  at 
(703)  614-6303  (DSN  224-6303). 


Part  I  -  Introduction 


Background 


Maintenance  and  diagnostic  automatic  test  equipment  is  used  to  support  the 
electronic  maintenance  testing  requirements  of  weapons  systems  at  the 
organizational,  intermediate,  and  depot  levels  of  maintenance.  The  automatic 
test  equipment  includes  test  program  sets  (primarily  computer  software  and 
hardware  interfaces)  used  to  adapt  automatic  test  equipment  systems  to  the 
individual  testing  requirements  of  specific  weapon  systems.  The  F-15 
Downsized  Tester  (the  Tester)  will  be  a  portable  automatic  test  equipment 
system  for  testing  a  portion  of  the  electronic  ^uipment  on  the  F-15  aircraft  at 
the  intermediate  level.  The  Tester  would  initially  replace  the  F-15  Avionics 
Intermediate  Shop  (AIS)  that  has  been  in  operation  since  1974.  The  Tester 
would  later  replace  the  F-15E  Mobile  Electronic  Test  Set  (METS)  used  on  the 
F-15E  model  aircraft  since  1986. 

In  1990,  the  Air  Force  attempted  to  procure  a  Radio  Frequency  METS  (a  larger 
modified  model  of  the  F-15E  METS)  to  replace  the  F-15  AIS  test  equipment 
and  to  provide  new  test  equipment  for  the  Special  Operations  Forces 
AC-130  Gunship.  During  the  acquisition,  Allied  Signal,  Inc.,  filed  a  protest  in 
U.S.  District  Court  of  Delaware  because  the  Air  Force  was  planning  a  sole- 
source  procurement  with  McDonnell  Douglas  Corporation.  The  U.S.  Distrirt 
Court  issued  a  stipulation  of  dismissal  r^uiring  that  the  Air  Force  notify  Allied 
Signal,  Inc.,  within  24  hours  of  a  decision  to  acquire  automatic  test  equipment 
through  other  than  full  and  open  competition. 

In  April  1992,  the  Air  Force  issued  to  contractors  a  request  for  cost  ^d 
technical  proposals  for  the  Tester.  The  Air  Force  began  its  source  selection 
process  for  the  development  of  the  Tester  in  June  1992. 

During  the  budgetary  process,  the  House  Appropriations  Committee  issued 
Report  102-627  on  June  29,  1992,  directing  the  Secretary  of  the  Air  Force  to 
establish  procedures  and  programs  to  standardize  test  equipment.  Ultimately, 
conferees  from  the  House  and  Senate  Appropriations  Committees  concluded  m 
the  House  of  Representatives  Conference  Report  102-1015,  October  5,  1992, 
that  the  Air  Force  could  not  afford  to  develop  test  equipment  unique  to  the  Air 
Force  or  to  weapon  systems.  As  an  interim  measure  to  DoD-wide  policy  on 
automatic  test  equipment  commonality.  Report  102-1015  required  the  Milit^ 
Departments  to  submit  a  cost  benefit  analysis  comparing  newly  recommended 
systems  acquisitions  to  the  Army  Integrated  Family  of  Test  Equipment  (IF'TO) 
and  the  Navy  Consolidated  Automated  Support  System  (CASS).  A  description 
of  the  IFTE  and  CASS  programs  is  in  Appendix  A.  The  restrictions  were 
applicable  to  all  weapons  systems  being  developed  or  modified,  except  for  the 
Tester  contract,  which  was  to  be  award^  in  the  near  future. 

The  Principal  Deputy  Assistant  Secretary  of  Defense  (Production  and  I^^stics) 
sent  a  memorandum  to  the  Assistant  Secretary  of  the  Air  Force  (Acquisition  on 
January  26,  1993,  stating,  "The  Air  Force  should  redirect  the  DST  [F-IS 
Downsized  Tester]  acquisition  to  use  the  IFTE  or  another  inventory  DoD 
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standard  ATE  [automatic  test  equipment-Navy's  CASS],  or  revise  its  acquisition 
approach  to  address  known  and  projected  DoD-wide  requirements  for  a 
downsized  tester."  The  Principal  Deputy  did  not  endorse  the  acquisition  of  the 
Tester  because  the  acquisition  did  not  comply  with  standardization  policy  for 
automatic  test  equipment.  The  Principal  Deputy  stated  that  the  Air  Force 
initiated  the  acquisition  without  a  current  analysis  of  the  existing  standard 
automatic  test  equipment,  without  a  review  of  other  upcoming  "downsized"  test 
requirements,  and  without  an  approved  operational  requirements  document. 

In  response  to  the  Principal  Deputy's  January  26,  1993,  memorandum,  the 
Deputy  Assistant  Secretary  of  the  Air  Force  (Acquisition)  issued  a  memorandum 
on  February  12,  1993,  that  provided  a  history  of  the  Tester  acquisition  program 
with  analyses  to  show  that  Air  Force  could  not  justify  the  redirection  of  the 
acquisition.  The  memorandum  concluded  by  stating,  "Unless  sp^ifically 
directed  to  the  contrary  by  the  DoD  Acquisition  Executive,  it  is  our  intent  to 
proceed  with  our  acquisition  strategy  to  acquire  the  F-15  DST  [F-15  Downsized 
Tester]  through  full  and  open  competition." 

The  Deputy  Assistant  Inspector  General  for  Auditing,  Office  of  the  Inspector 
General,  DoD,  sent  a  memorandum  on  March  22,  1993,  to  the  AssistMt 
Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller)  requesting 
that  the  Air  Force  suspend  the  source  selection  for  a  Government  contractor  to 
develop  the  Tester  until  after  the  audit  of  the  Tester  was  completed,  reviewed 
by  the  Air  Force,  and  any  disagreements  with  our  conclusions  were  resolved. 
Preliminary  results  indicated  that  the  Air  Force  issued  the  solicitation  to 
Government  contractors  without  adequate  justification  or  approval  of  the 
requirement  specifications  for  the  Tester  by  the  ^r  Force  Air  Combat 
Command  and  without  adequate  evaluations  of  existing  DoD  families  of 
standard  automatic  test  equipment. 

In  response  to  the  March  22,  1993,  Deputy  Assistant  Inspector  General  for 
Auditing  memorandum,  the  Deputy  Assistant  Secretary  of  the  Air  Force 
(Acquisition)  stated  on  April  5,  1993,  that  the  Air  Force  did  not  agree  with 
suspending  the  source  selection  until  the  completion  of  the  audit.  The  Deputy 
Assistant  Secretary  stated  that  although  the  draft  operational  requiremente 
document  was  not  approved  by  the  Air  Force  Chief  of  Staff,  the  Air  Force  Air 
Combat  Command  agreed  with  the  requirement  for  the  Tester.  Additionally, 
the  Deputy  Assistant  Secretary  stated  that  the.  Air  Force  adequately  evaluat^ 
the  DoD  families  of  standard  automatic  test  equipment.  However,  the  Air 
Force  did  agree  to  delay  the  award  of  the  contract  for  the  Tester  until  after  the 
audit  analysis  was  completed  and  the  Air  Force  had  reviewed  the  final  report. 


Objectives 


The  primary  objective  of  the  audit  was  to  evaluate  the  Air  Force's  acquisition  of 
unique  test  equipment  to  replace  existing  F-15  avionics  test  equipment. 
Specifically,  we  determined  if  the  Air  Force  complied  with  DoD  policies  and 
proc^ures  for  acquiring  automatic  test  equipment,  and  if  the  Air  Force  justified 
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the  acquisition  of  the  F-15  Downsized  Tester  by  fully  documenting  requirements 
and  evaluating  alternative  test  equipment  in  the  Army  and  Navy.  Additionally, 
we  evaluated  the  effectiveness  of  the  applicable  internal  controls. 


Scope  of  Audit 


We  evaluated  requirements  documents,  contractual  records,  cost  benefit 
analyses,  system  studies,  and  other  program  documentation  covering  the  period 
from  June  1988  through  March  1993  that  pertained  to  the  acquisition  of  the 
Tester.  Additionally,  we  evaluated  the  Air  Force’s  compliance  with  DoD 
general  policies  and  procedures  for  standardizing  automatic  test  equipment.  We 
also  reviewed  the  adequacy  of  program  budgeting  for  the  proposed  Tester. 

We  discussed  the  reasonableness  of  the  acquisition  stotegy  for  the  Tester  with 
senior  management  officials  in  the  Office  of  the  Assistant  Secretary  of  Defense 
(Production  and  Logistics);  the  Office  of  the  Assistant  Secretary  of  the  iMr 
Force  (Acquisition);  the  Air  Force  Commander,  San  Antonio  Air  Logistics 
Center;  and  the  Air  Force  Program  Manager  and  Procurement  Contracting 
Officer  for  the  Tester.  This  economy  and  efficiency  audit  was  made  fi-om 
February  through  May  1993,  in  accordance  with  auditing  standards  issued  by 
the  Comptroller  General  of  the  United  States,  as  implemented  by  the  Inspector 
General,  DoD,  and  accordingly  included  such  tests  of  internal  controls  as  were 
considered  necessary.  Organizations  visited  or  contacted  during  the  audit  are 
listed  in  Appendix  D. 


Internal  Controls 


The  audit  identified  material  internal  control  weaknesses  as  defined  by  Public 
Law  97-255,  Office  of  Management  and  Budget  Circular  A-123,  and  DoD 
Directive  5010.38.  Controls  were  not  effective  to  ensure  that  automatic  test 
equipment  being  acquired  for  the  F-15  aircraft  was  properly  justified  and  cost- 
effective.  Internal  control  weaknesses  are  discussed  in  detail  in  the  Finding  and 
controls  assessed  are  discussed  in  Part  1.  Recommendations  1.  and  2.,  if 
implemented,  will  assist  in  correcting  the  weaknesses.  Undeterminable  savings 
will  be  achieved  through  standardization  of  equipment  and  support  from 
implementing  the  recommendations.  A  copy  of  this  final  report  will  be 
provided  to  the  internal  control  officers  within  the  Office  of  the  Secretary  of 
Defense  (OSD)  and  the  Air  Force. 
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Prior  Audits 


We  issued  two  audit  reports  in  1992  addressing  deficiencies  in  OSD  Md  Air 
Force  policies  and  procedures  on  standardizing  automatic  test  equipment. 
Inspector  General  (IG),  DoD,  Report  No.  92-037,  "Effectiveness  of  the  Air 
Force's  Internal  Controls  over  the  Development  and  Acquisition  of  Maintenance 
and  Diagnostic  Systems,"  January  23,  1992,  stated  that  Air  Force  commands 
were  not  complying  with  Air  Force  guidance  for  acquiring  standardized 
automatic  test  equipment.  We  recommended  that  the  Air  Force  approve 
developments  of  new  automatic  test  equipment  only  when  it  can  be 
demonstrated  that  it  would  not  be  cost-effective  to  acquire  the  standard 
automatic  test  equipment  developed  under  the  Army's  IFTE  and  the  Navy's 
CASS  programs.  The  Air  Force  did  not  specifically  address  the 
recommendation  in  its  comments  to  the  audit  report.  In  a  June  17,  1992, 
memorandum  to  the  Deputy  Assistant  Inspector  General  (Inspection,  General 
Accounting  Office,  and  Audit  Followup),  the  Assistant  Secretary  of  the  Air 
Force  agreed  that  the  Air  Force  would  promulgate  policy  requiring  that 
selection  of  automated  test  equipment  would  be  based  on  a  cost  benefit  analysis 
of  equipment  or  system  alternatives  (including  software)  available  in  the  DoD 
inventory. 

IG,  DoD,  Report  No.  92-095,  "Acquisition  and  Management  of  Maintenance 
and  Diagnostic  Automatic  Test  ^uipment,"  May  21,  1992,  reported  that 
management  deficiencies  by  the  Military  Departments  and  the  lack  of  uniform 
and  comprehensive  DoD-wide  policy  and  guidance  contributed  to  the  continued 
proliferation  of  maintenance  and  diagnostic  automatic  test  equipment  and 
seriously  affected  the  cost-effectiveness  of  acquisitions.  We  recommended  that 
comprehensive  and  uniform  DoD-wide  policy  and  guidance  on  the  acquisition 
and  management  of  maintenance  and  diagnostic  automatic  test  equipment  be 
developed  and  implemented.  The  Assistant  Secretary  of  Defense  (Production 
and  Logistics)  agreed  in  principle  with  our  recommendation  and  performed  a 
DoD-wide  automatic  test  systems  investment  strategy  study  that  was  presented 
to  Congress  on  May  20,  1993.  The  study  will  iirovide  a  basis  for  DoD 
automatic  test  systems  policy  guidelines  and  assignment  of  management 
responsibilities. 


Other  Matters  of  Interest 


During  our  discussions  with  the  Air  Force  management  officials,  the  Air  Force 
raised  concerns  about  congressional,  legal,  and  cost  implications  caused  by  any 
changes  in  the  acquisition  strategy  for  the  Tester.  The  Air  Force's  concerns  are 
unwarranted. 

The  Air  Force  asserted  that  Congress  made  clear  its  intention  in  House 
Conference  Report  102-1015  that  the  Tester  program  was  to  proceed  in 
accordance  with  the  Air  Force's  competitive  acquisition  strategy.  Although 
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the  House  Conference  Report  excused  the  Air  Force  from  submitting  a  cost 
benefit  analysis  for  the  Tester  to  the  congressional  defense  committees,  we 
believe  the  Report  did  not  preclude  the  Air  Force  from  preparing  a  cost  benefit 
analysis  for  the  Secretary  of  Defense. 

In  House  Conference  Report  102-1015,  the  conferees  concluded  that  the  Air 
Force  could  not  afford  to  develop  test  equipment  unique  to  the  Milit^ 
Departments  or  weapon  systems.  As  an  interim  measure  to  DoD-wide  policy 
on  automatic  test  equipment  commonality,  the  House  Conference  Report 
required  the  Military  Departments  to  submit  a  cost  benefit  analysis  comparing 
newly  recommended  systems  acquisitions  to  the  Army's  IFTE  and  the  Navy's 
CASS.  The  House  Conference  Report  states  that  restrictions  are  applicable  to 
all  weapons  systems  being  develop^  or  modified,  except  the  Tester.  Taken  in 
context,  the  congressional  language  does  not  appear  to  restrict  the  Secretary  of 
Defense's  authority  to  redirect  the  program  if  defense  acquisition  management 
policies  and  procedures  were  not  followed. 

In  its  February  12,  1993,  response  to  the  Principal  Deputy  Assistant  Secretary 
of  Defense  (Production  and  Logistics),  the  Air  Force  stated  that  purchasing  the 
Army's  IFTE  would  reverse  the  Air  Force's  present  acquisition  strategy  for  full 
and  open  competition.  Through  our  discussions  with  Air  Force  officials  in  the 
contracting,  competition  advocate,  and  general  counsel  offices,  we  were  unable 
to  determine  any  obstacle  to  purchasing  the  Army  test  equipment  if  thorough 
analysis  justified  such  a  decision  to  proceed  with  other  than  full  and  open 
competition. 

First,  the  request  for  proposal  in  the  full  and  open  competition  for  the  Tester 
was  not  based  on  acquiring  standardized  automatic  test  equipment,  but  was 
prepared  for  the  unique  testing  requirements  of  some  of  the  avionics  subsystems 
on  the  F-15  aircraft.  Requirements  and  contract  specifications  for  st^dardiz^ 
automatic  test  equipment  would  be  broader  in  scope  and  include  additional  Air 
Force  avionics  subsystems  and  other  aircraft,  as  well  as  the  potential  for  other 
DoD  weapon  systems.  As  a  result  of  identifying  additional  requirements,  the 
basis  for  determining  full  and  open  competition  would  be  different.  Therefore, 
sole-source  procurement  of  the  Army  and  Navy's  test  equipment  could  be 
justifiable  partially  on  the  basis  that  theirs  is  the  only  standardized  test  systems 
in  production  in  the  current  DoD  inventory. 

Second,  the  Army  is  planning  to  have  a  full  and  open  competition  for  the 
follow-on  purchase  of  the  IFTE  after  FY  1995.  If  sole  sourcing  presents  a 
significant  obstacle,  the  Air  Force  could  delay  its  acquisition  until  FY  1996. 
During  the  interim  period,  the  Air  Force  could  continue  to  use  existing 
equipment  and  complete  thorough  cost  and  requirements  analyses  and  provide 
new  contract  specifications  for  the  Army's  follow-on  purchase. 

Finally,  awarding  a  sole-source  contract  in  light  of  the  U.S.  District  Court  of 
Delaware  stipulation  for  dismissal  does  have  a  precedent.  Although  the  Air 
Force's  acquisition  of  a  downsized  tester  for  the  AC-130  Gunship  was  affected 
by  the  Delaware  Court  ruling,  the  Special  Operations  Forces  later  processed  a 
justification  and  approval  for  a  sole-source  procurement  of  new  automatic  test 
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equipment  for  the  AC-130  Gunship.  When  Allied  Signal,  Inc.,  filed  another 
protest,  the  General  Accounting  Office  ruled  in  favor  of  the  Air  Force  and  the 
SpeciaJ  Operations  Forces. 

Air  Force  officials  stated  that  the  procurement  process  for  the  Tester  needed  to 
be  accelerated  to  avoid  undue  schedule  slippage  and  costs  for  acquiring  new  test 
equipment.  The  Air  Force  program  management  office  contract^  with  ARINC 
Research  Corporation  to  prepare  a  study,  "F-15  AIS  Supportability  Study,"  that 
compared  the  cost  of  buying  new  automatic  test  equipment  to  the  cost  of 
upgrading  and  modifying  the  F-15  AIS  test  equipment.  According  to  the 
ARINC  study,  over  91  percent  of  the  $458  million  needed  for  the  F-15  AIS 
tester  was  for  modernizing,  upgrading,  and  purchasing  repair  parts  for  the  test 
equipment  and  F-15  aircraft  through  the  year  2010.  After  discussions  with 
ARINC  and  the  San  Antonio  Program  Management  Office  representatives,  we 
concluded  that  prior  to  the  discontinuance  of  the  upgrade  the  Air  Force  had 
already  bought  the  repair  parts  for  the  test  ^uipment.  We  also  concluded  that 
the  annual  cost  for  repairing  the  test  equipment  is  about  $75,000.  If  the 
purchase  of  new  test  equipment  was  delayed  for  2  years  (the  amount  of  time 
estimated  by  the  Air  Force  to  perform  sufficient  a,nalyses  and  prepare  contract 
specifications,  if  necessary),  it  would  cost  the  Air  Force  $150,000  to  repair 
existing  automatic  test  equipment.  The  Tester  action  officer  at  the  Air  Force 
Air  Combat  Command  stated  that  Bendix  Corporation  was  committed  to 
supporting  the  F-15  AIS  equipment  as  long  as  necessary. 

If  the  Air  Force  does  not  defer  the  replacement  of  the  F-15  AIS  tester,  it  would 
not  get  the  full  utility  of  the  investment  made  for  modernizing  the  primary  test 
equipment  and  upgrading  the  test  stations,  including  the  cost  for  repair  parts  that 
were  stocked.  Although  the  Air  Force  made  an  investment  in  the  F-15  AIS 
tester  by  modernizing,  upgrading,  and  purchasing  repair  parts  that  extend^  the 
usefulness  of  the  computer  test  sets  through  the  year  2010,  the  Air  Force  did  not 
consider  this  investment  during  the  preparation  of  the  contract  specifications  for 
the  Tester.  As  a  result,  the  Tester  may  be  replacing  more  components  of  the 
F-15  AIS  tester  than  is  necessary. 
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Part  II  -  Finding  and  Recommendations 


Acquisition  of  F-IS  Downsized  Tester 

The  Air  Force  was  planning  to  develop  and  procure  unique  automatic 
test  equipment  that  is  not  cost-effective.  The  condition  occurred  because 
the  Air  Force  did  not  comply  with  defense  acquisition  management 
guidance  and  perform  sufficient  cost,  technical,  and  requirements 
analyses  to  determine  if  standard  DoD  families  of  test  equipment  or 
other  automatic  test  equipment  could  satisfy  Air  Force  and  other  DoD 
test  equipment  requirements.  Additionally,  OSD^and  the  Air  Force  had 
not  developed  and  implemented  effective  policy  and  guidance  on 
standardized  automatic  test  equipment.  As  a  result,  the  Air  Force  could 
spend  $167  million  for  55  units  of  the  F-15  Downsized  Tester  over  the 
6-year  Future  Years  Defense  Program  (FY  1992  through  FY  1997)  with 
capabilities  limited  to  testing  portions  of  the  F-15  avionics  systems. 


Background 


The  Air  Force  initiated  the  acquisition  of  unique  automatic  test  equipment  for 
the  F-15  aircraft  in  1990  that  would  test  portions  of  the  aircraft’s  avionics 
systems.  Standard  automatic  test  equipment,  such  as  the  Army's  IFTE  and  the 
Navy's  CASS,  were  entering  the  DoD  inventory  during  this  same  period.  The 
standard  automatic  test  equipment  programs  are  more  cost-effective  because 
they  have  capabilities  for  testing  multiple  electronic  or  avionics  systems  and  can 
be  used  to  support  several  weapons  systems.  The  Air  Force  pursued  the 
acquisition  of  the  Tester  without  thoroughly  evaluating  the  Army's  IFTE  and 
the  Navy's  CASS  as  potentially  more  cost-effective  alternatives.  We  attributed 
the  Air  Force's  inadequate  evaluations  to  noncompliance  with  defense 
acquisition  management  guidance  and  the  lack  of  OSD  and  Air  Force 
standardization  policy. 


Defense  Acquisition  Management 


The  Air  Force  did  not  comply  with  defense  acquisition  management  policies  and 
procedures  for  the  acquisition  of  the  Tester.  DoD  Directive  5000.1,  "Defense 
Acquisition,"  and  DoD  Instruction  5000.2,  "Defense  Acquisition  Management 
Policies  and  Procedures,"  both  dated  February  23,  1991,  established  a 
disciplined  management  approach  for  acquiring  major  and  nonmajor  systems 
and  materiel  that  satisfy  the  operational  users'  needs.  The  guidance  identifies 
core  activities  that  must  be  accomplished  for  every  acquisition  program,  such  as 
a  documented  operational  requirement  and  a  cost  and  operational  effectiveness 
analysis.  One  management  principle  in  the  guidance  is  that  a  full  range  of 
alternatives  must  be  considered  before  deciding  to  initiate  a  new  acquisition 
program.  The  preferred  or  first  alternative,  in  an  order  of  precedence,  is  that  a 
DoD  Component  use  or  modify  an  existing  U.S.  military  system  to  satisfy  an 
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operational  requirement.  Although  the  Army's  IFTE  or  the  Navy's  CASS  were 
viable  alternatives,  the  Air  Force  did  not  evaluate  the  Army  or  Navy's  programs 
before  initiating  the  acquisition  of  the  Tester  in  April  1992. 

Analysis  of  Army  and  Navy's  Automatic  Test  Equipment. 
The  Air  Force  did  not  implement  recommendations  in  Report  No.  92-037  to 
approve  developments  of  new  automatic  test  equipment  only  when  it  could  be 
demonstrated  that  it  would  not  be  cost-effective  to  acquire  the  standard 
automatic  test  equipment  developed  under  the  Army's  IFTE  and  the  Navy  s 
CASS.  In  June  1992,  5  months  after  our  fin^  report  was  issued,  the  Air  Force 
agreed  that  it  would  promulgate  policy  requiring  that  selection  of  automated  test 
equipment  would  be  based  on  a  cost  benefit  analysis  of  equipment  or  system 
alternatives  (including  software)  available  in  the  DoD  inventory.  However, 
before  the  June  1992  agreement,  the  Air  Force  disregarded  our  recommendation 
and  in  April  1992  it  issued  a  request  for  technical  and  cost  proposes  for  me 
Tester  without  evaluating  the  Army  or  Navy’s  systems.  In  July  1992,  the  Air 
Force  initiated  a  . review  of  the  Army  and  Navy's  systems  only  after  the  House 
Appropriations  Committee  directed  the  Secretoty  of  the  Air  Force  to  estabhsh 
procedures  and  programs  to  standardize  test  equipment  for  Air  Force  systems. 

Air  Force  Review  of  Automatic  Test  Equipment.  The  Air 
Force's  review  of  automatic  test  equipment  was  conducted  in  two  phas^. 
Phase  1  was  to  survey  all  Air  Force  programs  to  identify  current  or  evolving 
automatic  test  equipment  requirements  at  the  intermediate,  depot,  and  factory 
levels  of  maintenance  and  to  determine  the  status  of  automatic  test  equipment 
efforts.  Phase  2  was  to  be  a  detailed  cost  and  technical  assessment  of 
appropriate  Air  Force  programs  identified  in  Phase  1  compared  to  the  Army's 
IFTE  and  the  Navy's  CASS.  The  Air  Force  accelerated  the  Phase  2  cost 
assessment  for  the  Tester  and  included  preliminary  results  of  the  assessment  in 
the  Phase  1  report.  The  Air  Force  issued  a  final  report  for  Phase  1  on 
September  11,  1992,  and  a  final  report  on  Phase  2  on  March  31,  1993.  The 
final  report  on  Phase  1  did  not  ad^uately  address  the  cost  and  technical 
comparisons  needed  to  assess  standardization  of  automatic  test  equipment  for  the 
Tester. 


Cost  Assessment.  The  Air  Force  cost  assessments  for  the 
Tester  cannot  be  relied  upon  for  significant  acquisition  decisions  because  the 
assessments  were  incomplete  and  were  not  properly  prepared  and  validated. 
Two  cost  assessments  that  the  Air  Force  completed,  the  ARINC  Res^ch 
Corporation  cost  study  and  the  cost  assessment  reported  in  the  Phase  1  final 
report,  did  not  satisfy  the  requirement  for  a  cost  and  operational  effectiveness 
analysis. 

DoD  Instruction  5000.2  states  that  a  cost  and  operational  effectiveness  analysis 
should  be  prepared  early  in  the  acquisition  process  for  systems  acquisitions  and 
approved  by  the  Air  Force  Acquisition  Executive.  The  cost  and  operational 
effectiveness  analysis  is  intended  to  illuminate  the  advantages  and  disadvantages 
of  alternative  systems  being  considered  for  acquisition  and  the  sensitivity  of 
each  alternative  to  possible  changes  in  assumptions  or  variables  such  as  select^ 
performance  capabilities.  A  cost  and  operational  effectiveness  analysis  will 
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draw  on  several  subanalyses,  including  analyses  of  mission  needs,  the  threat  and 
U.S.  capabilities,  the  interrelationship  of  systems,  the  contribution  of  multi-role 
systems,  measures  of  effectiveness,  costs,  and  cost-effectiveness  comparisons. 

Instead  of  a  cost  and  operational  effectiveness  analysis,  the  Air  Force  program 
office  at  the  San  Antonio  Air  Logistics  Center  initially  contracted  with  ARINC 
Research  Corporation  for  a  study  that  identified  cost  savings  that  would  be 
realized  from  purchasing  new  automatic  test  equipment  versus  extending  the  life 
of  the  F-15  AIS  tester.  The  ARINC  cost  study  was  completed  in  June  1991  and 
was  the  only  cost  study  initially  used  to  justify  the  acquisition  of  the  Tester. 

After  receiving  the  congressional  direction  and  after  starting  the  source  selection 
process  in  its  full  and  open  competition,  the  Air  Force  accelerated  its  Phase  2 
analysis  of  the  Tester  and  included  a  cost  assessment  of  the  Tester  in  its  Phase  1 
survey  of  automatic  test  equipment.  The  Air  Force  Aeronautical  Systems 
Center  at  Wright-Patterson  Air  Force  Base,  Ohio,  completed  a  cost  assessment 
on  September  11,  1992.  The  assessment  consisted  of  composite  estimates  of 
contractor  proposals  for  the  Tester  compared  to  estimated  costs  for  Army  s 
IFTE  and  Navy’s  CASS.  It  also  included  technical  data  on  the  performance  of 
the  Army  IFTE  and  Navy  CASS  compared  to  the  performance  requirements  in 
the  draft  operational  requirements  document  for  the  Tester.  Air  Force  officMs 
stated  that  the  assessment  relied  on  cost  and  technical  data  solicited  from  the 
Army  IFTE  and  Navy  CASS  program  offices  as  well  as  the  Air  Force  program 
office  for  the  Tester.  However,  the  Air  Force  officials  were  unable  to  provide 
adequate  evidence  to  show  that  the  Army  and  Navy  program  offices  verified  the 
cost  and  technical  data  used  in  the  Air  Force's  cost  assessment.  Additionally, 
officials  at  the  Air  Force  Materiel  Command  did  not  approve  the  cost 
assessment  as  prescribed  by  Air  Force  Logistics  Command  Supplement  1  to  Air 
Force  Regulation  173-1,  "The  Air  Force  Cost  Analysis  Program,"  June  15, 
1989.  Although  the  Aeronautical  Systems  Center  had  not  completed  the  process 
of  updating  and  approving  the  cost  assessment  for  the  Tester,  the  San  Antonio 
program  office  was  concurrently  evaluating  the  best  and  final  offers  from 
contractors  in  its  full  and  open  competition  for  the  Tester.  The  Air  Force’s 
actions  imply  that  the  Air  Force  had  littte  or  no  intention  of  changing  its 
acquisition  strategy  regardless  of  the  outcome  of  the  cost  assessment  of  the 
Tester  compared  to  the  Army  and  Navy’s  test  equipment. 

The  Air  Force’s  cost  study  and  cost  assessment  were  not  adequate  substitutes  for 
the  cost  and  operational  effectiveness  analysis  required  by  defense  acquisition 
management  guidance  and  Air  Force  implementing  guidance,  Air  Force 
Regulation  57-1,  "Air  Force  Mission  Needs  and  Operational  Requirements 
Process,"  August  1,  1992.  The  cost  study  and  cost  assessment  were  inadequate 
because  they  did  not  include  evaluations  of  cost,  performance,  and  schedule 
tradeoffs  for  differences  in  the  alternative  Army  and  Navy  standardize 
automatic  test  equipment,  as  required  by  DoD  guidance.  For  example,  the  Air 
Force's  technical  assessment,  discussed  later  in  this  report,  showed  variances  in 
performance  characteristics  for  the  Army’s  IFTE.  Although  the  variances  were 
the  bases  for  disqualifying  the  Army  test  equipment  as  a  viable  alternative,  the 
Air  Force  did  not  evaluate  tradeoffs  that  could  be  made  in  the  cost, 
performance,  or  schedule  of  the  test  equipment  and  the  ultimate  impact,  if  any, 
on  the  mission  success  of  the  F-15  aircraft  as  a  result  of  the  variances. 
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The  Air  Force's  cost  assessment  and  comparison  of  the  Army's  IFTE  and  the 
Navy's  CASS  also  did  not  evaluate  the  benefits  resulting  from  standardizing  test 
equipment.  The  essence  of  standardizing,  as  in  the  Army  and  Navy's  automated 
test  systems,  is  that  cost  benefits  are  derived  from  having  one  piece  of  test 
equipment  to  test  electronic  subsystems,  modules,  or  components  from  several 
different  avionics  subsystems,  such  as  electronic  warfare,  communications,  and 
radar  equipment,  from  severk  different  weapon  systems,  such  as  F-15,  F-16, 
and  F-111  aircraft  in  a  composite  Air  Force  wing.  Cost  savings  accrue  from 
having  to  purchase  and  provide  maintenance  support  for  only  one  test  system 
instead  of  several  different  test  systems.  A  draft  OSD  investment  strategy 
report  on  automatic  test  systems  stated  that  an  average  cost  savings  of  25  to 
35  percent  could  result  from  reductions  in  development  costs,  economies  of 
scale,  and  workload  efficiencies  through  standardization  of  equipment.  The 
Tester  that  the  Air  Force  is  competing  among  several  contractors  will  have 
projected  life  cycle  costs  of  $269  million  and  will  test  only  some  subsystems, 
electronic  modules,  and  components  from  the  F-15  aircraft.  The  Army  and 
Navy  test  systems  can  be  modified  to  test  electronic  modules  from  several 
avionics  subsystems  and  several  different  aircraft  to  include  the  F-15  aircraft. 

The  Air  Force's  cost  assessment  did  not  adequately  compare  and  evaluate  the 
operating  and  support  costs  of  the  alternative  Army  and  Navy  standard 
automatic  test  equipment.  Results  from  OSD's  yearlong  investment  strategy 
study,  which  was  provided  to  Congress  on  May  20,  1993,  support  our 
conclusion  that  the  Air  Force  had  insufficient  cost  data  to  compare  and  evaluate 
the  Tester  with  the  Army  and  Navy’s  systems.  The  OSD  study  concluded  that 
costs  of  automatic  test  equipment  gathered  and  maintained  by  the  Military 
Departments  consisted  primarily  of  hardware  development  and  procurement 
costs.  According  to  the  OSD  study,  there  were  insufficient  data  available  in  the 
Military  Departments  to  assess  operating  and  support  costs  including  software, 
which  can  be  a  dominant  cost  factor.  Insufficient  operating  and  support  costs 
data  are  evident  in  the  Air  Force  Aeronautical  Systems  Center's  cost  assessment 
because  the  development,  production,  operating,  and  support  costs  for  the  test 
program  sets  were  shown  as  the  same  for  each  system.  Without  complete  data, 
Air  Force  system  developers  cannot  compare  and  evaluate  life  cycle  costs  for 
each  alternative  automatic  test  equipment. 

Technical  Assessment.  The  Air  Force  perform^ 
insufficient  analyses  of  performance  data  and  requirements  for  new  automatic 
test  equipment  to  support  its  technical  assessment.  The  Air  Force  program 
management  office  at  San  Antonio,  Texas,  completed  a  technical  assessment  on 
December  22,  1992,  that  compared  the  technical  parameters  of  the  Army's 
IFTE  with  the  technical  requirements  for  the  Tester.  Based  on  the 
December  22,  1992,  technical  assessment,  the  Air  Force  concluded  that  the 
Army's  IFTE  could  not  satisfy  the  performance  and  deployment  requirements 
for  testing  electronic  equipment  on  the  F-15  aircraft.  Specifically,  the  Army 
test  equipment's  mean  time  between  failure  in  hours  was  too  low,  the  Army's 
test  equipment  could  not  be  carried  on  two  pallets  and  setup  by 
four  maintenance  technicians,  and  the  Army's  test  equipment  had  technick 
operating  deficiencies,  such  as  the  inability  to  detect  faults  in  electronic  modules 
within  specified  parameters. 
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The  Air  Force  did  not  perform  sufficient  analyses  to  support  the  time 
between  failure  technic^  requirements  in  the  draft  operational  requirements 
document  for  the  Tester,  The  draft  operational  requirements  document  stated 
that  the  new  test  equipment  was  required  to  have  a  mean  time  between  failure  of 
1,000  hours.  Officials  at  the  Air  Force  Air  Combat  Command  coordinating  the 
draft  operational  requirements  document  could  not  provide  any  documentation 
or  other  analyses  to  show  the  need  for  the  specific  performance  criterion,  as 
required  by  Air  Force  Pamphlet  57-1.  Operations  personnel  indicated  that 
performance  data  on  the  mean  time  between  failure  were  not  being  collected  and 
analyzed  because  the  data  collecting  capability  for  the  maintenance  of  test 
equipment  was  unreliable.  Therefore,  the  performance  of  existing  and  future 
test  equipment  could  not  be  measured  and  evaluated  accurately. 

The  1,000-hour  mean  time  between  failure  criterion  had  doubled  over  the 
criterion  cited  in  a  previous  requirements  document  approved  in  1988,  which 
was  referred  to  in  the  draft  operational  requirements  document  as  the  source 
document.  Officials  at  the  Air  Force  Air  Combat  Command  did  not  have  a 
audit  trail  between  the  two  documents,  as  required  by  Air  Force  Pamphlet  57-1, 
and  could  not  explain  why  the  mean  time  between  failure  for  the  new  test 
equipment  increased  from  500  hours  in  1988  to  1,000  hours  in  1993.  The  Air 
Force  disqualiried  the  Army's  IFTE  in  its  December  22,  1992,  technical 
assessment  on  the  technical  basis  that  the  Army’s  test  equipment  could  not  meet 
the  1,000-hour  mean  time  between  failure  requirement  for  the  test  equipment. 
However,  the  mean  time  between  failure  performance  specification  for  the 
Army  test  equipment  of  590  hours  would  have  met  the  1988  Air  Force 
performance  requirement  of  500  hours. 

We  noted  similar  discrepancies  in  the  Air  Force's  analyses  supporting  other 
performance  requirements  in  its  draft  operational  requirements  document; 
including  the  ne^  for  two  pallet  transportability,  four-mM  setup  and  operation, 
95-percent  system  availability,  and  requirements  to  isolate  faults  in  shop 
repairable  electronic  units  ranging  from  95  to  100  percent.  According  to  Air 
Force  Pamphlet  57-1,  the  operational  requirements  document  should  contain  a 
requirements  correlation  matrix  that  cites  system  operational  characteristics  and 
capabilities  that  are  realistic,  meaningful,  and  germane  to  the  mission  need. 
Values  for  each  threshold  and  objective  must  be  described  based  on  their 
relationship  to  mission  success  and  how  the  initial  values  were  selected.  The 
Army  program  manager  for  the  IFTE  implied  that  the  Air  Force  systems' 
parameters  were  unrealistic,  in  his  conclusion;  and  that  the  Air  Force  system 
mean  time  between  failure  was  significantly  higher  than  the  mean  time  between 
failure  experienced  for  similar  test  equipment  in  the  Army.  Additionally,  the 
Air  Force  Air  Combat  Command  could  not  provide  data  analyses  showing  how 
the  values  or  parameters  were  selected,  the  cost  effect  of  the  increased  values, 
and  their  relationship  to  mission  success. 

Regarding  technical  values  and  parameters,  the  Air  Force  had  inconsistencies  in 
its  evaluations  of  cost  and  technical  data  as  performed  by  personnel  at  the  San 
Antonio  Air  Logistics  Center  and  the  Aeronautical  Systems  Center.  For 
example,  the  program  manager  at  San  Antonio  found  that  the  70-percent  system 
availability  and  98-percent  fault  detection  parameters  for  the  Army's  IFTE  were 
unacceptable  in  comparison  to  the  respective  95  and  100  percent  parameters 
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specified  in  the  draft  operational  requirements  document.  However,  personnel 
at  the  Aeronautical  Systems  Center  performing  the  cost  and  technical  analyses 
report^  that  the  Army  test  equipment  performance  criteria  were  acceptable. 
Without  adequate  data  analysis,  a  complete  audit  trail  for  the  Air  Force's 
performance  requirements  in  its  draft  operational  requirements  document,  and 
with  inconsistent  evaluations  of  technical  parameters,  the  Air  Fotccs 
disqualification  of  the  Army  test  equipment  on  a  technical  basis  was  unjustified. 

Operational  Requirements  Document  Approval.  The 

Air  Force  had  no  official  basis  for  establishing  the  technical  criteria  for 
evaluating  alternatives  to  the  Tester  or  for  preparing  contract  specifications  for  a 
full  and  open  competition.  Defense  acquisition  management  guidance  states  that 
the  operational  user  prepares  an  operational  requirements  document  to  provide  a 
basis  for  system-specific  performance  requirements  written  in  the  contract 
specifications.  The  Air  Force  began  processing  its  draft  operational 
requirements  document  for  the  Tester  in  October  1991.  The  program  manager 
for  the  Tester  developed  and  issued  a  draft  request  for  proposal  to  Government 
contractors  in  January  1992  and  issued  a  final  request  for  propos^  in 
April  1992.  Although  the  Air  Force  had  not  coordinated  the  draft  operational 
requirements  document  for  final  approval  until  a  year  later  on  March  31, 
it  had  planned  to  award  a  contract  for  the  development  and  procurement  of  the 
Testers  as  early  as  March  1,  1993.  The  developing,  processing,  and  approving 
of  an  operational  requirements  document  is  out  of  sequence  with  sound 
acquisition  management  procedures.  The  operational  requirements  document 
should  have  been  prepared  and  approved  before  request  for  proposals  were 
issued  in  January  1992. 


Policy  on  Standardized  Automatic  Test  Equipment 


Neither  OSD  nor  the  Air  Force  had  developed  and  implemented  effective  policy 
and  guidance  on  the  standardization  of  automatic  test  equipment.  The  Assistant 
Secretary  of  Defense  (Production  and  Logistics)  chartered  a  study  on 
February  7,  1992,  to  develop  a  DoD  investment  strategy  for  automatic  test 
equipment.  A  study  team  of  OSD  and  Military  Department  officials  conducted 
an  extensive  review  from  February  through  August  1992  of  automatic  test 
equipment  acquisition  costs,  policies,  processes,  organizations,  and  technical 
needs  and  capabilities.  Although  completed  in  August  1992,  OSD  had  not 
officially  finalized  the  results  of  the  study. 

In  the  October  5,  1992,  House  Conference  Report  102-1015,  Congress  directed 
the  Secretary  of  Defense  to  develop  a  DoD-wide  policy  requiring  automatic  test 
equipment  commonality  in  standards  among  the  Military  Departments  along 
with  an  oversight  system  to  ensure  compliance.  As  of  June  1,  1993,  OSD  had 
not  issued  specific,  comprehensive  policy  on  the  acquisition  and  oversight  of 
automatic  test  equipment  in  DoD.  OSD  did  include  general  guidance  in  the 
revised  DoD  Directive  4151.18,  "Maintenance  of  Military  Materiel," 
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August  12,  1992,  that  test  measurement  and  diagnostic  equipment,  including 
automatic  test  systems  and  their  associated  software  programs,  shall  be 
standardized  among  DoD  Components. 

Congress  also  directed  the  Secretary  of  Defense  to  ensure  that  the  Air  Force 
establish  a  Secretarial  mandate  and  control  procedures  for  standardized  test 
equipment.  As  of  June  1,  1993,  the  Air  Force  had  no  specific,  comprehensive 
policy  on  the  acquisition  of  automatic  test  equipment.  However,  the  Air  Force 
was  establishing  an  organization  at  the  San  Antonio  Air  Logistics  Center  to 
establish  policies  and  procedures  on  standardized  automatic  test  equipment. 

In  summary,  the  Air  Force  did  not  have  the  organization,  guidance,  and  data 
bases  necessary  to  evaluate  and  monitor  the  acquisition  of  automatic  test 
equipment  to  ensure  that  equipment  would  satisfy  multiple  requirements  of 
electronic  subsystems  and  weapon  systems.  The  acquisition  of  the  Tester 
evolved  in  the  absence  of  adequate  OSD  and  Air  Force  policy  and  guidance  on 
standardized  automatic  test  equipment. 


Summary 


The  primary  concern  of  the  Director,  Weapon  Support  Improvement  Group,  is 
that  the  Air  Force  is  not  developing  a  family  of  standardized  automatic  test 
equipment,  or  using  existing  families  in  DoD,  that  meet  multiple  weapon  system 
and  cross  Military  Department  test  requirements.  We  agree  with  the  intent  of 
the  Principal  Deputy  Assistant  Secretary  of  Defense  (Production  and  Logistics) 
to  standardize  automatic  test  equipment  in  DoD  when  he  advised  the  Air  Force 
to  redirect  its  acquisition  of  the  Tester.  The  Principal  Deputy  had  legitimate 
concerns  about  the  requirements  and  the  cost  benefits  of  the  Tester  because  the 
Air  Force  did  not  comply  with  procedures  and  practices  for  ensuring  the  cost- 
effective  acquisition  of  automatic  test  equipment.  Additionally,  the  Air  Force’s 
uncertainties  concerning  the  effects  of  congressional,  legal,  cost,  schedule,  and 
policy  issues,  as  discussed  in  Other  Matters  of  Interest,  do  not  preclude  the  Air 
Force  from  deferring  the  acquisition  of  the  Tester  pending  sufficient  cost, 
technical,  and  requirements  analyses.  Therefore,  Ae  Air  Force  could  avoid 
spending  $167  million  for  unique  automatic  test  equipment  for  the  F-15  mrcraft 
by  redirecting  its  acquisition  strategy  to  procure  standardized  automatic  test 
equipment. 


Recommendations,  Management  Comments,  and  Audit 
Response 


1.  We  recommend  that  the  Under  Secretary  of  Defense  (Acquisition  and 
Technology)  develop  and  implement  uniform  and  comprehensive  DoD-wide 
policy  and  guidance  on  the  acquisition  and  management  of  standardized 
automatic  test  equipment. 
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Management  Comments.  The  Principal  Deputy  Assistant  Secretary  of 
Defense  (Production  and  Logistics)  concurred  with  Recommendation  L,  stating 
that  draft  DoD  automatic  test  systems  acquisition  policy  and  guidance  was  being 
prepared  and  coordinated  with  the  Services.  The  Principal  Deputy  Assistant 
Secretary  also  stated  that  the  corrective  action  would  be  more  appropriate  for 
the  Under  Secretary  of  Defense  (Acquisition). 

Audit  Response.  We  revised  Recommendation  1.  by  addressing  the 
recommendation  to  the  Under  Secretary  of  Defense  (Acquisition  and 

Technology).  We  request  that  the  Under  Secretary  comment  on  the  revised 
Recommendation  1.  in  responding  to  the  final  report.  The  comments  should 
include  an  estimated  completion  date  for  implementing  DoD-wide  policy  and 
guidance. 

2.  We  recommend  that  the  Assistant  Secretary  of  the  Air  Force  (Acquisition): 

a.  Discontinue  the  full  and  open  competition  acquisition  of  the  F-15 
Downsized  Tester. 

Management  Conunents.  The  Deputy  Assistant  Secretary  of  the  Air  Force 
(Acquisition)  nonconcurred  with  Recommendation  2.a.,  and  the  potential 
monetary  benefits  associated  with  the  recommendation,  stating  that 

discontinuing  the  full  and  open  competition  of  the  Tester  would  result  in 
monetary  losses.  The  monetary  losses  would  include  costs  for  the  maintenance 
and  repair  of  the  F-15  AIS  tester  and  increased  market  prices  for  a  replacement 

tester,  or  increased  costs  for  the  Army’s  IFTE  and  costs  to  reimburse 

contractors  for  preparing  their  bids  on  the  Tester  if  a  contract  is  not  awarded. 

Regarding  the  potential  monetary  benefits,  the  Deputy  Assistant  Secretaty  stated 
that  the  2-year  delay  in  the  acquisition  of  the  Tester  could  cost  the  Air  Force 
over  $245  million  in  continued  support  of  the  aging  F-15  AIS  tester  and  the 
eventual  acquisition  of  a  replacement  test  system  at  FY  1996  and  FY  1997 
market  prices.  Potential  lawsuits  by  the  Tester  bidders  could  result  in  a,dditional 
monetary  losses.  Procurement  of  the  IFTE  would  cost  the  Air  Force 
approximately  $215  million,  as  opposed  to  the  $167  million  for  the  Tester.  The 
complete  text  of  the  Air  Force  comments  are  included  in  Part  IV.  Air  Force 
comments  to  the  finding  and  our  response  to  those  comments  are  in 
Appendix  B. 

Audit  Response.  The  potential  monetary  losses  that  the  Air  Force  cited  from 
discontinuing  the  acquisition  of  the  Tester  are  a  product  of  the  Air  Force's 
improper  planning  and  imprudent  management  decisions.  Had  the  Air  Force 
followed  defense  acquisition  management  guidance  it  could  have  avoided  delays 
in  the  acquisition,  and  the  delays'  associated  costs,  by  performing  adequate  cost, 
technical,  and  requirements  analyses  before  the  Air  Force  issued  the  requests  for 
proposals  to  Government  contractors.  We  believe  the  Air  Force  could  have 
exercised  better  management  judgment  when  the  Princijjal  Assistant  Secretary  of 
Defense  (Production  and  Logistics)  requested  the  Air  Force  to  redirect  its 
acquisition  strategy.  After  the  Principal  Deputy  Secretary's  request  and  after 
we  began  our  requested  audit,  the  Air  Force  released  its  request  for  best  and 
final  offers  from  the  Government  contractors,  although  there  was  no  apparent 
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urgency  and  management  differences  on  the  acquisition  strategy  were 
unresolved.  With  this  action,  the  Air  Force  not  only  increased  the  prosp^Uve 
costs  to  Government  contractors  for  preparing  potentially  unnecessary  bids  but 
the  risks  for  liability  to  the  Government  to  pay  the  costs  were  dso  increased.  It 
the  Air  Force  should  incur  some  monetary  losses  by  redirecting  its  acquisition 
strategy,  we  believe  benefits  from  developing  and  acquiring  standardized 
automatic  test  equipment  would  still  exce^  any  monetary  losses  incurrM 
because  of  the  Air  Force's  improper  planning.  Therefore,  we  request  that  the 
Assistant  Secretary  of  the  Air  Force  (Acquisition)  reconsider  its  position  in 
response  to  the  final  rqiort. 

Concerning  the  potential  monetary  benefits,  the  Air  Force  did  not  provide 
documentation  to  support  its  claim  that  $245  million  was  needed  for  the 
maintenance  of  the  F-15  AIS  tester  for  2  years  and  the  cost  of  its  replacement. 
The  only  analysis  the  Air  Force  provided  to  us  was  the  AWNC  Research 
Corporation  study,  which  showed  that  the  maintenance  and  repair  costs  for  the 
F-15  AIS  would  be  $150,000  for  2  years.  We  continue  to  believe  that  the  Air 
Force  needs  to  perform  new  analyses  for  acquisition  of  standardize  automatic 
test  equipment  that  includes  the  complete  F-15  AIS,  addition^  requirements  Iot 
other  aircraft  beyond  the  F-15,  and  potential  test  capabilities  for  other  Dc^ 
weapon  systems.  Until  new  analyses  are  prepared  and  properly  approved,  the 
$167  million  programmed  for  a  new  Tester  is  unjustified  and  should  not  be 
spent  on  the  current  acquisition.  We  have  modified  the  report  to  indicate  that 
net  potential  monetary  benefits  are  currently  undeterminable.  We  request  that 
the  Assistant  Secretary  of  the  Air  Force  (Acquisition)  reconsider  its  position  on 
the  estimat^  monetary  benefits  in  the  response  to  the  final  report. 

b.  Prepare  a  revised  operational  requirements  document  prescribed  by 
Air  Force  Pamphlet  57-1  that  is  based  on  a  thorough  and  supportable  analysis  of 
realistic  and  cost-effective  user  performance  requirements.  The  operational 
requirements  document  should  be  expanded  in  scope  to  include  performan^ 
requirements  for  standardized  automatic  test  equipment.  It  should  also  be 
approved  by  the  Air  Force  Chief  of  Staff. 

Management  Comments.  The  Deputy  Assistant  Secretary  of  the  Air  For^ 
(Acquisition)  nonconcurred  with  Recommendation  2.b.,  stating  that  the 
operational  requirements  document  for  the  Tester  was  approved  by  the  Air 
Force  Chief  of  Staff  on  May  11,  1993.  The  Deputy  Assistant  Secretary  also 
stated  that  the  need  for  replacing  the  F-15  AIS  tester  was  identified  years  ago 
and  support  for  the  F-15  weapon  system  cannot  be  delayed  indefinitely  while 
requirements  are  continually  reevaluated. 

Audit  Response.  Although  the  operational  requirements  document  was 
approved,  we  were  not  provided  a  copy  of  the  approved  document  and  ml 
supporting  analyses  for  our  review.  Therefore,  we  continue  to  believe  that  the 
requirements  (such  as  the  1,000  hour  mean  time  between  failure  requirement 
cited  in  the  document  were  not  adequately  justified  with  complete  and  thorough 
analyses  that  should  have  been  done  when  the  need  to  replace  the  F'fo 
tester  was  first  identified.  We  also  believe  that,  in  this  case,  a  delay  is 
appropriate  for  identifying  realistic  requirements  to  avoid  purchasing  expCTSiye 
automatic  test  equipment  with  limited  testing  capabilities.  The  Air  Force  should 
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evaluate  additional  automatic  test  equipment  requirements  for  other  weapon 
systems  in  its  inventory  and  develop  new  performance  criteria  for  acquiring 
standardized  automatic  test  equipment  such  as  the  Army's  IFTE  or  Navy's 
CASS.  We  request  that  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
reconsider  its  position  in  response  to  the  final  report. 

c.  Prepare  a  cost  and  operational  effectiveness  analysis  that  cornpares 
the  Army's  Integrated  Family  of  Test  Equipment  and  the  Navy's  Consolidated 
Automated  Support  System  to  the  new  requirements  developed  for  standardized 
automatic  test  equipment.  The  cost  and  operational  effectiveness  should  also 
include  an  evaluation  of  the  improvements  that  have  been  made  to  existing 
automatic  test  equipment  in  determining  the  incremental  need  for  new  test 
equipment  and  test  program  sets. 

Management  Comments.  The  Deputy  Assistant  Secretary  of  the  Air  Force 
(Acquisition)  nonconcurred  with  Recommendation  2.c.,  stating  that  a  cost  and 
operational  effectiveness  analysis  is  required  for  acquisition  category  (ACAT)  I 
programs  only  and  that  authority  for  the  designation  of  ACAT  II,  III,  and  IV 
programs  is  delegated  to  the  Service  Acquisition  Executive.  The  Air  Force 
decided  that  it  would  not  be  cost-effective  to  accomplish  a  full  cost  and 
operational  effectiveness  analysis  for  this  program  when  several  cost  analyses 
have  already  been  accomplished. 

Audit  Response.  Although  the  cost  and  operational  effectiveness  analysis  is 
mandatory  only  for  ACAT  I  programs,  the  underlying  principles  and  anmytical 
concepts  are  to  be  tailored  and  implemented  in  support  of  ACAT  H,  III,  and  IV 
programs.  As  stated  in  DoD  Instruction  5000.2,  the  cost  and  operational 
effectiveness  analysis  is  one  of  the  required  core  activities  that  must  be 
accomplished  for  every  acquisition.  The  Air  Force  did  not  tailor  and  implement 
the  principles  of  a  cost  and  operational  effectiveness  analysis  in  the  cost  analyses 
discussed  in  the  report.  A  cost  and  operational  effectiveness  analysis  is  still 
needed  to  evaluate  the  potential  tradeoffs  in  cost,  performance,  and  schedule 
among  the  alternative  programs,  and  fully  evaluate  the  benefits  of 
standardization  of  automatic  test  equipment.  We  request  that  the  Air  Force 
provide  us  the  newly  referenced  documentation  and  reconsider  its  position  in 
response  to  the  final  report. 

d.  Develop  and  implement  comprehensive  Air  Force  policy  and 
procedures  on  the  standardization  of  automatic  test  equipment. 

Management  Comments.  The  Deputy  Assistant  Secretary  of  the  Air  Force 
(Acquisition)  concurred  with  Recommendation  2.d.,  stating  that  policy  for  the 
acquisition  of  automatic  test  equipment  would  be  issued  by  June  30,  1993. 

Audit  Response.  The  Air  Force's  action  is  responsive.  As  part  of  the  follow¬ 
up  process,  we  request  a  copy  of  the  policy  when  the  Air  Force  provides 
comments  to  the  final  report. 
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Appendix  A,  Description  of  the  Army's 

Integrated  FamUy  of  Test 
Equipment  and  the  Navy's 
Consolidated  Automated  Support 
System 


The  Army's  IFTE  is  composed  of  five  subsystems  of  automatic  test  equipment. 
The  primary  intermediate  maintenance  level  subsystem  is  the  Base  Shop  Test 
Facility  (the  Test  Facility)  which  is  installed  in  an  S-280  shelter  and  mounted  on 
a  5-ton  truck  for  Army  use.  The  Test  Facility  is  general  purpose,  multi¬ 
functional  automatic  test  equipment  that  uses  several  configurations  of  tests  to 
diagnose  electronic  faults  or  failures  in  weapon  systems.  The  Test  Facility 
entered  low  rate  initial  production  in  February  1989  and  full  scale  production  in 
March  1992.  The  Army  has  authorized  the  acquisition  of  121  Test  Facilities  to 
support  26  weapon  systems  by  FY  1998.  The  Army  is  ordering  57  Test 
Facilities  for  Army  weapon  systems,  including  the  AVENGER  ^d  HAWK 
missile  systems;  the  All  Source  An^ysis  System;  and  the  Multiple  Launch 
Rocket  System,  that  will  be  put  into  operation  throughout  FY  1993. 

The  Navy's  CASS  is  automatic  test  equipment  composed  of  six  electronic 
configurations:  the  Hybrid  Station;  Electro-Optical  Station;  Radar  Station; 
Electronic  Warfare  Station;  Communication,  Navigation,  and  Identification 
Station;  and  Digital  Display  Station.  The  CASS  is  general  purpose,  multi¬ 
functional  automatic  test  equipment  for  diagnosing  electronic  faults  or  failures 
in  weapon  systems  that  is  used  at  the  intermediate  maintenance  levels,  primarily 
aboard  aircraft  carriers,  and  depot  maintenance  levels.  The  CASS  entered  the 
engineering  and  manufacturing  development  phase  in  December  1986  and  full 
scale  engineering  in  1987.  The  CASS  is  currently  in  the  second  low  rate  initial 
production  phase  and  full  rate  production  is  expected  in  the  second  quarter  of 
FY  1994.  The  Navy  is  planning  to  use  the  CASS  in  support  of  60  weapon 
systems  at  1 19  different  sites. 
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Appendix  B:  Audit  Responses  to  Specific  Air 

Force  Comments 

The  following  paragraphs  provide  audit  responses  to  specific  Air  Force 
comments. 


Prior  Audit  Coverage 


Air  Force  Conunents.  Page  2,  para  2;  The  Air  Force  stated  that  our  history  5 
was  incomplete  regarding  IG,  DoD,  Audit  Report  No.  92-037  and  the  resolution 
process.  The  Air  Force  provided  additional  details. 

Audit  Response.  The  prior  audit  coverage  comments  in  the  draft  audit  report 
were  not  meant  to  be  comprehensive  but  were  a  synopsis  of  salient  points 
pertinent  to  the  audit  finding. 


Internal  Controls 


Air  Force  Comments.  Page  3,  para  2:  The  Air  Force  did  not  agree  with  our  4 

statement  that  controls  were  not  effective  to  ensure  that  automatic  test 
equipment  being  acquired  for  the  F-15  aircraft  was  properly  justified  and  cost- 
effective,  stating  that  the  acquisition  is  justified  and  is  the  most  cost-effective 
solution  for  this  requirement. 

Audit  Response.  In  other  comments  to  the  draft  report  the  Air  Force  agreed 
that  it  did  not  comply  with  defense  acquisition  management  policies  and 
procedures,  that  it  did  not  have  data  analyses  available  to  support  the 
operational  requirements  document,  and  that  it  did  not  implement  specific, 
comprehensive  policy  on  automatic  test  equipment.  The  above  are  some  of  the 
internal  controls  needed  to  ensure  that  automatic  test  equipment  is  properly 
justified  and  cost-effective.  Therefore,  we  still  believe  that  our  statement  is 
applicable. 


Background 


Air  Force  Comments.  Page  4,  para  3:  The  Air  Force  disageed  with  our  2 
statement  that  the  House  Appropriations  Committee  Report  102-1015  required  a 
cost  benefit  analysis  comparing  newly  recommended  systems  acquisitions  to  the 
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Army  IFTE  and  the  Navy  CASS.  The  Air  Force  stated  that  the  wst  benefit 
an^ysis  was  to  include  a  comparison  of  the  recommended  systems  with  existing 
systems  within  the  DoD,  the  IFTE  or  CASS  system,  a  modified  IFTE  or  CASS 
system,  and  modular  components  of  existing  systems.  Report  102-1015  also 
stated  that  the  restrictions  on  automatic  test  equipment  apply  to  all  weapons 
systems  being  developed  and  to  modifications  for  upgrading  existing  avionics 
systems,  except  the  Tester  contract,  which  is  to  be  awarded  in  the  near  future. 

Regarding  the  statement  on  page  5,  para  2,  the  Air  Force  noted  that  the 
February  12,  1993,  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
memorandum  replying  to  Assistant  Secre^  of  Defense  (Production  and 
Logistics)  provided  a  history  of  this  acquisition  program  that  included  analyses 
of  the  Army’s  IFTE  and  Federal  District  Court  litigatiori  over  a  sole-source 
attempt  to  procure  automatic  test  equipment  in  the  DoD  inventory.  The  Air 
Force  could  not  justify  redirection  of  the  acquisition  to  a  sole-source  contract 
program,  such  as  the  Army's  IFTE  because  adequate  competition  existed. 

Audit  Response.  We  agree  that  the  scope  of  the  congressional  language  w^ 
broader.  We  paraphrased  the  information  we  believed  to  be  relevant  to  this 
report.  Concerning  Air  Force  comments  on  its  response  to  the  Principal  Deputy 
Assistant  Secretary  of  Defense  (Production  and  Logistics),  we  revised  the 
background  for  clarity. 


Discussion 


Air  Force  Comments.  Page  6,  para  1:  Although  the  Air  Force  concurred  with  lo 

our  statement,  "The  Air  Force  did  not  comply  with  defense  acquisition 
management  policies  and  procedures  for  the  acquisition  of  the  tester... a  full 
range  of  alternatives  must  be  considered  before  deciding  to  initiate  a  new 
acquisition  program,"  the  Air  Force  stated  that  a  range  of  ^ternative  automatic 
test  systems  were,  in  fact,  considered  and  the  Air  Force  originally  attempted  to 
procure  an  inventoried  system,  the  METS. 

Audit  Response.  We  disagree  that  the  Air  Force  adequately  considered  a  full 
range  of  automatic  test  equipment.  The  Air  Force  provided  no  documentation 
to  support  that  any  automatic  test  equipment,  such  as  Ae  IFTE  was  fully 
evaluated  before  it  issued  the  requests  for  proposals.  Air  Force  acquisition 
officials  stated  that  no  additional  analyses  were  necessary  because  the  F-15 
METS  performed  well  in  Operation  Desert  Shield  and  Desert  Storm  and  the  Air 
Force  users  wanted  that  equipment.  The  Air  Force  ignored  the  suggestion  in 
the  1988  High  Mobility  Tester  Statement  of  Need  to  evaluate  the  Army's  IFTE 
and  Navy's  CASS  as  viable  alternatives.  Instrad,  the  Air  Force  attempted  to 
make  a  sole-source  procurement  of  an  F-15  Radio  Frequency  METS.  When  the 
Air  Force  could  no  longer  pursue  the  acquisition  of  a  radio  frequency  METS, 
officials  stated  that  requirements  were  prepared  for  the  Tester  request  for 
proposal  that  were  based  on  the  METS  performance  characteristics.  We  do  not 
believe  the  actions  by  the  Air  Force  were  consistent  with  sound  acquisition 
management  policy. 
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Air  Force  Comments.  Page  6,  para  2:  The  Air  Force  disagreed  with  our  ii 
statement,  "The  Air  Force  disregarded  our  recommendation  and  in  April  1992  it 
issued  a  request  for  technical  and  cost  proposals  for  the  Tester  without 
evaluating  the  Army  or  Navy  systems,"  stating  that  the  solicitation  process  had 
already  begun  when  audit  Report  No.  92-037  was  issued.  The  draft  solicitation 
was  released  for  industry  comments  in  early  February  1992,  prior  to  receipt  of 
audit  Report  No.  92-037.  Further,  the  Air  Force  was  nei^er  directed  nor 
requested  to  hold  any  of  its  automatic  test  equipment  acquisition  programs  in 
abeyance  pending  the  mediation  of  this  report. 

Audit  Response.  Air  Force  acquisition  officials  were  made  aware  of  the 
viability  of  alternative  Army  and  Navy  automatic  test  equipment  before  the 
source  selection  process  began  for  the  Tester.  Report  No.  92-037  merely 
reinforced  what  the  Air  Force's  1988  High  Mobility  Tester  Statement  of  Need 
suggested  -  that  the  Army  IFTE  and  the  Navy  CASS  were  viable  alternative 
automatic  test  equipment  to  replace  the  F-15  AIS  tester.  We  believe  that 
evaluating  these  alternatives  before  it  issued  the  requests  for  proposals  in  April 
1992  potentially  committing  the  Government  to  pay  bid  and  proposal  costs) 
would  have  been  the  prudent  or  responsible  management  decision  for  the  Air 
Force  to  make,  without  being  direct^  or  requested  to  hold  or  stop  any  specific 
acquisition. 

Air  Force  Comments.  Page  7,  para  1:  The  Air  Force  nonconcurred  with  the  n 
draft  report  statement,  "The  Air  Force  cost  assessments  for  the  Tester  cannot  be 
relied  upon  for  significant  acquisition  decisions  bwause  the  assessments  were 
incomplete  and  were  not  properly  prepared  and  validated.  .  .a  COEA  [cost  ^d 
operational  effectiveness  an^ysis]  should  be  prepared  early  in  the  acquisition 
process.  .  .and  approved  by  the  Air  Force  Acquisition  Executive."  The  Air 
Force  stated  that  defense  acquisition  management  guidance  establishes  a  firm 
requirement  for  a  cost  and  operational  effectiveness  analysis  for  ACAT  I 
programs  only.  The  Tester  was  an  ACAT  IV  program  that  did  not  require  a 
cost  and  operational  effectiveness  analysis.  The  Air  Force  further  stated  that  it 
recognized  that  a  cost  analysis  should  be  performed  and  noted  five  studies  that 
were  completed  prior  to  the  release  of  the  request  for  proposals  for  the  Tester. 

The  Air  Force  concluded  that,  collectively,  the  five  studies  provided 
management  with  the  tools  necessary  to  make  informed  decisions  and  a  selection 
of  alternatives. 

Audit  Response.  Although  the  cost  and  operational  effectiveness  an^ysis  is 
mandatory  only  for  acquisition  category  I  programs,  the  underlying  principles 
and  analytical  concepts  are  to  be  tailored  and  implement^  in  support  of 
acquisition  category  II,  III,  and  IV  programs.  DoD  Instruction  5000.2  states 
that  there  are  core  activities  that  must  be  accomplished  for  every  acquisition 
program.  "These  core  activities  establish  and  document  the  threat  and 
operational  requirements,  affordability,  the  acquisition  strategy  and  program 
baseline,  cost  and  operational  effectiveness,  production  readiness  and 
supportability,  and  developmental  and  operational  testing.  .  .Tailoring  shall 
focus  on  how  these  activities  are  conducted,  the  formality  of  reviews  and 
documentation,  and  the  need  for  other  supporting  activities."  The  Air  Force  did 
not  accomplish  the  cost  and  operational  effectiveness  of  the  Tester  and  did  not 
tailor  its  acquisition  strategy.  The  program  management  office  representatives 
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stated  that  instructions  they  received  in  managing  the  acquisition  were  oral  and 
evolutionary.  The  Air  Force  did  not  provide  four  of  the  five  studies  mentioned 
in  its  comments,  but,  the  cost  assessments  that  the  Air  Force  provided  the  audit 
staff  did  not  satisfy  the  requirements  of  a  cost  and  operational  effectiveness 
analysis  for  reasons  discussed  in  the  report.  We  soil  believe  a  cost  and 
operational  effectiveness  analysis  is  the  appropriate  cost  analysis  as  required  in 
defense  acquisition  management  guidance. 

Air  Force  Comments.  Page  8,  para  1:  The  Air  Force  nonconcurred  with  our  12 

statement,  "Air  Force  officials  were  unable  to  provide  adequate  evidence  to 
show  that  the  Army  and  Navy  program  offices  verified  the  cost  and  technical 
data  used  in  the  Air  Force's  cost  assessment,"  stating  that  at  the  time  of  the  IG, 

DoD,  visit  to  Aeronautical  Systems  Center,  the  cost  assessment  was  not 
complete  and  formal  Army  and  Navy  coordination  was  not  yet  appropriate.  The 
Army  and  Navy  had  assigned  members  to  the  assessment  team,  who  provided 
Army  and  Navy  cost  figures.  The  Air  Force  has  coordinated  the  final  report  in 
writing  with  the  Army  on  the  IFTE  and  verbally  with  the  Navy  on  the  CASS. 

Audit  Response.  Our  comments  were  based  on  the  Phase  I  cost  assessment 
completed  on  September  11,  1992.  The  Army  and  Navy  did  participate  in  the 
cost  assessment  by  providing  cost  data;  however,  our  contacts  with  the  Army 
representatives  did  not  result  in  an  agreement  with  the  conclusions  reached  by 
the  Air  Force  officials  using  the  cost  data.  In  March  1993,  the  Army  responded 
to  an  Air  Force  special  request  for  the  Army  to  review  the  Air  Force's 
comparative  studies  of  the  Tester  and  Army's  IFTE.  The  Army  responded  that 
the  IFTE  was  a  potentially  attractive  alternative  to  Ae  Tester  and  that 
requirements,  cost,  and  leg^  concerns  about  the  acquisition  appeared  to  be 
caused  by  inconsistencies  in  Air  Force  stotegies  and  not  by  operational 
shortfalls  of  the  equipment.  The  Army  implied  that  the  Air  Force  cost 
assessment  was  incomplete  because  the  cost  assessment  did  not  compare  the 
IFTE  to  the  total  inventory  of  Air  Force  intermediate  maintenance  level  titers 
to  capture  the  significant  cost  and  operational  benefits  of  the  multi-functional 
systems. 

Air  Force  Comments.  Page  8,  para  1:  The  Air  Force  disagreed  with  our  12 

statement,  "The  Air  Force's  actions  imply  that  the  Air  Force  had  little  or  no 

intention  of  changing  its  acquisition  strategy  regardless  of  the  outcome  of  the 

cost  assessment  of  the  tester  compared  to  the  Army  and  Npy's  test  equipment," 

stating  that  the  Air  Force  was  in  source  selection  at  the  time  of  the  review  and 

had  neither  been  directed  nor  requested  to  stop  the  process.  The  program  was 

not  even  questioned  by  OSD  until  October  28,  1992,  in  a  request  for  a  briefing. 

Audit  Response.  See  our  comments  under  Air  Force  comments,  page  6, 
para  2. 

Air  Force  Comments.  Page  8,  para  2:  The  Air  Force  nonconcurred  with  our  12 

statement,  "The  Air  Force's  cost  study  and  cost  assessment  were  not  adequate 
substitutes  for  the  cost  and  operational  effectiveness  analysis... The  cost 
assessment  was  inadequate  because  it  did  not  include  evaluations  of  cost, 
performance,  and  schedule  tradeoffs  for  differences  in  the  alternative  Army  and 
Navy  standardized  automatic  test  equipment,  as  required  by  DoD  guidance," 
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stating  that  the  cost  assessment  accomplished  by  Aeronautic^  Systems  Center 
was  not  intended  to  be  a  substitute  for  a  cost  and  operational  effectiveness 
analysis,  but  a  comparative  cost  analysis  of  the  Tester  versus  adapting  the 
Army's  IFTE  and  Navy's  CASS.  The  analysis  was  part  of  a  comprehensive  Air 
Force  review  that  examined  all  automatic  test  equipment  acquisition  programs  to 
determine  the  technical  and  programmatic  impacts  of  acc^uiring  IFTE  or  CASS. 

The  analysis  process  was  validated  by  the  Air  Force  Audit  Agency  on  March  5, 

1993.  The  Air  Force  further  stated  that  the  an^ysis  included  benefits  of 
standardi2atioh  such  as  economies  of  scale,  reductions  in  support  equipment, 
and  training  courses  and  software  tools  that  have  been  purchased  already.  Other 
potential  standardization  savings  were  not  evaluated  because  Air  Force  did  not 
have  a  requirement  for  an  intermediate  maintenance  level  tester  at  a  composite 
wing. 

Audit  Response.  For  reasons  stated  in  other  sections  of  this  report,  we  believe 
that  Air  Force  cost  analyses  were  inadequate  and  the  cost  and  operational 
effectiveness  analysis  is  appropriate  for  the  Tester  acquisition. 

The  Air  Force  Audit  Agency  validated  that  the  economic  analysis  model 
prescribed  in  Air  Force  regulations  was  appropriately  applied  for  the  cost 
analysis  of  an  avionics  subsystem  for  an  unspecified  aircraft.  The  Audit 
Agency's  review  did  not  determine  if  the  appropriate  model  was  chosen  for  the 
Tester,  did  not  verify  any  cost  data  for  the  Tester  or  Army  and  Navy  test 
equipment,  and  did  not  consider  the  effects  of  standardization. 

The  benefits  of  standardization  that  the  Air  Force  may  have  included  in  the  F- 15 
cost  assessment  did  not  address  the  much  larger  savings  that  can  be  achieved 
through  standardization.  For  example,  the  operating  system  software  will  not 
have  to  be  developed  for  seven  different  testers  if  a  standardized  tester  is 
developed.  Likewise,  training  courses,  calibration  equipment,  and  maintenance 
programs  will  not  have  to  be  developed  and  procured  for  seven  different 
programs.  The  OSD  automatic  test  equipment  investment  strategy  study 
concluded  that  an  average  cost  savings  of  30  percent  could  result  from 
standardization,  which  amounts  to  over  $178  million  for  the  seven  programs 
included  in  the  Air  Force's  Phase  11  cost  assessment. 

Air  Force  Comments.  Page  9,  para  1:  The  Air  Force  nonconcurred  with  our  13 
statement,  "The  tester  that  the  Air  Force  is  competing  among  several  contractors 
will  have  projected  life  cycle  costs  of  $269  million  and  will  test  only  some 
subsystems,  electronic  modules,  and  components  from  the  F-15  aircraft," 
stating  that  while  the  Air  Force  agrees  that  the  IFTE  and  CASS  can  be  modified 
to  test  line  replaceable  units  from  the  F-15,  the  proposed  Tester  can  be  modified 
to  test  systems  other  than  the  F-15.  Unlike  the  man-portable  Tester,  IFTE  and 
CASS  should  require  further  modifications  to  meet  the  F-15's  deployment 
requirements.  It  should  be  noted  that,  although  the  Aeronautical  Systems 
Center  assessment  projects  a  life  cycle  cost  of  $269  million  for  the  proposed 
Tester,  it  also  projects  $351  million  for  CASS  and  $302  million  for  IFTE  for 
the  F-15  requirement,  assuming  no  slippages  to  the  current  program's  schedule. 
Slippage  to  the  current  program  schedule  to  accommodate  IFTE  would  cost  an 
additional  amount  of  approximately  $13.5  million  per  year  in  maintenance  costs 
for  continued  support  of  the  F-15  AIS. 
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Audit  Response.  The  Air  Force  was  unable  to  provide  any  documented 
support  to  show  that  systems  other  than  the  F-15  systems  specified  in  the 
request  for  proposal  could  be  tested  by  the  Tester. 

Based  on  our  analysis  of  die  ARINC  supportability  study,  only  $75,000  is 
needed  annually  for  the  maintenance  of  the  F-15  AIS  tester.  The  fife  cycle  cost 
of  die  Tester  is  not  comparable  to  the  life  cycle  cost  of  the  IFTE  or  the  CASS 
because  standardization  was  not  considered  in  the  one-on-one  compmsons.  The 
IFTE  and  CASS  have  many  more  potential  weapon  system  applications  than  the 
Air  Force  Tester.  Also,  the  life  cycle  costs  of  the  IFTE  and  CASS  are  more 
realistic  because  these  systems  are  becoming  operational  and  the  Tester  has  not 
been  developed.  See  our  audit  response  under  Air  Force  comments,  page  8, 
para  2,  for  potential  average  cost  savings  that  could  result  from  standardization. 

Air  Force  Comments.  Page  9,  para  2:  The  Air  Force  nonconcurred  with  our  13 

statement,  "The  Air  Force's  cost  assessment  did  not  adequately  compare  and 

ev^uate  the  operating  and  support  costs  of  the  alternative  Army  and  Navy  test 

equipment. "  It  stated  that  the  operating  and  support  cost  area  was  thoroughly 

examined  by  the  Automatic  Test  Equipment  Acquisition  Review  team  and 

changes  continued  to  be  made  after  the  auditors'  visit  to  Aeronautical  System 

Center.  Operating  and  support  costs  for  the  testers  and  test  program  sets  were 

also  considered.  Both  the  Army  and  Navy  provided  tester  operating  and  support 

cost  estimates  based  on  projections  for  organic  repair  costs  for  their  automatic 

test  equipment.  ITie  assessment  team  used  an  average  of  the  Army  and  Navy 

cost  estimates,  which  was  applied  to  the  Tester  as  a  percentage  of  the  automatic 

test  equipment's  production  costs  to  account  for  differences  in  complexity  of  the 

testers.  Because  there  was  very  little  cost  data  available  on  the  test  program  sets 

operating  and  support  costs,  the  assessment  team  used  an  average  of  Navy  and 

Air  Force  cost  data,  which  was  applied  as  a  fixed  percentage  of  test  program  set 

acquisition  costs  for  all  alternatives.  Test  program  set  development  and 

production  costs  were  also  estimated  as  the  same  for  afi  alternatives  because 

these  costs  vary  little  among  testers  with  the  same  capabilities. 

Audit  Response.  The  Air  Force  did  not  provide  the  audit  staff  a  complete  copy 
of  the  final  Phase  II  cost  assessment.  However,  we  determined  from  segments 
of  a  draft  of  the  Phase  11  cost  assessment  that  the  methodology  for  comparing 
the  Tester  operating  and  support  costs  changed  from  Phase  I  to  Phase  II.  For 
example,  the  operating  and  support  costs  for  the  Tester  in  Phase  I  were 
computed  as  five  percent  of  the  Tester  production  costs.  The  Air  Force 
provided  no  supporting  rationale  or  source  data  for  determining  the  five  percent 
except  that  the  percentage  was  purportedly  from  the  OSD  Investment  Strategy 
Study.  In  the  draft  results  of  Phase  II,  the  Air  Force  used  the  average  operating 
and  support  costs  of  the  Army's  IFTE  and  the  Navy's  CASS  to  determine  the 
percentage.  We  could  not  corroborate  any  of  the  cost  estimates  the  Air  Force 
made  to  the  source  cost  data  used  to  develop  the  percentages  for  the  estimates. 

We  do  not  believe  that  using  the  average  of  the  competing  Army  and  Navy 
general  purpose  test  equipment  costs  is  appropriate.  We  believe  the  Air  Force 
should  have  evaluated  the  costs  of  similar  Air  Force  system  specific  testers, 
such  as  METS,  for  applicability  to  the  Tester.  Instead  it  used  arbitrary 
estimates  applied  equally  for  all  the  test  equipment  evaluated. 
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Air  Force  Conunents.  Page  10,  para  1:  The  Air  Force  nonconcurred  with  the  13 
statement,  "The  Air  Force  performed  insufficient  analyses  of  performance  data 
and  requirements  for  new  automatic  test  equipment  to  support  its  requirements 
assessment,"  stating  that  the  San  Antonio  Air  Logistics  Center  Technical 
Assessment,  December  22,  1992,  compared  the  F-15  line  replaceable  units 
performance  requirements  to  the  available  stimuli, .  measurement,  and  power 
capacities  of  the  IFTE  system.  IFTE  operational  shortfalls  penning  to  mean 
time  between  failure,  transport  configuration,  and  portability  were  noted. 
Additionally,  fault  isolation  shortfalls,  human  factors  limitations,  operational 
deficiencies,  and  necessary  modifications  for  the  IFTE  system  were  identified. 

Audit  Response.  Our  report  refers  to  the  lack  of  analyses  that  the  Air  Force 
Air  Combat  Command  performed  to  support  the  requirements  or  performance 
data  stated  in  the  dr^t  operational  requirements  document.  The  performance 
data  were  used  as  the  basis  for  the  technical  assessment.  We  believe  the 
conclusions  of  the  technical  assessment  were  based  on  unsupported  requirements 
such  as  the  need  for  a  1,000-hour  mean  time  between  failure  and  two  man 
transportability. 

Our  report  also  noted  deficiencies  in  the  San  Antonio  Air  Logistics  Center 
technical  assessment  of  the  Army's  test  equipment.  The  technical  assessment: 

o  did  not  attempt  to  evaluate  the  impact  of  identified  variances  between 
the  Army's  IFTE  performance  statistics  and  the  Air  Force's  operational 
requirements  document, 

o  did  not  attempt  to  evaluate  the  impact  of  the  schedule  delay  associated 
with  the  modifications  of  Army  test  equipment, 

o  did  not  consider  the  completed  upgrades  to  the  F-15  AIS  tester, 

o  was  not  reviewed  by  the  Army  test  equipment  engineers,  who  later 
disagreed  with  the  conclusions  of  the  assessment,  and 

0  was  inconsistent  with  some  of  the  Aeronautical  Systems  Center 
engineers'  technical  assessments. 

Air  Force  Comments.  Page  10,  para  2:  The  Air  Force  nonconcurred  with  our  i4 
statement,  "The  Air  Force  did  not  perform  sufficient  analyses  to  support  mean 
time  between  failure  technical  requirements  in  the  draft  operational  requirements 
document,"  stating  that  experience  with  the  F-15  METS  during  System 
Performance  Demonstration  produced  an  831-hour  mean  time  between  failure 
for  a  similar  device.  The  1,000  hour  mean  time  between  f^ure  requirement 
grew  out  of  the  Air  Force’s  Reliability  and  Maintenance  2000  initiative. 

Audit  Response.  The  Air  Force  provided  no  information  or  documentation  on 
the  F-15  METS  System  Performance  Demonstration  or  the  Air  Force's 
Reliability  and  Maintainability  2000  initiative  during  the  audit.  The  data 
provided  to  the  audit  staff  from  the  F-15  System  Program  Office  showed  that 
METS  specifications  were  500  hours  for  the  mean  time  between  failure  and  that 
METS  demonstrated  a  performance  of  588  hours.  We  believe  this  actual  data 
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of  an  operational  tester  could  be  considered  realistic  and  meaningful,  which  is 
consistent  with  the  guidelines  of  Air  Force  Pamphlet  57-1,  However,  the  dato 
do  not  indicate  whether  the  characteristics  are  germane  to  the  mission  need.  Air 
Force  system  developers  should  take  full  advantage  of  the  technology  in  the 
market  place  whenever  that  technology  is  practical  and  cost-effective. 

However,  the  presumed  availability  of  the  technology  should  not  be  the  sole 
basis  for  the  requirement.  We  stiU  believe  that  the  Air  Force  did  not 
demonstrate  that  the  1,000-hour  mean  time  between  failure  was  realistic, 
meaningful,  and  critical  to  the  mission  success  of  the  F-15  aircraft. 

Air  Force  Comments.  Page  10,  para  2:  The  Air  Force  disagreed  with  our  14 
statement  that,  "Officials  at  Air  Force  Air  Combat  Command.  .  .could  not 
provide  any  documentation  or  other  analysis  to  show  the  need  for  specific 
performance  criterion... personnel  indicated  that  such  performance  data  were  not 
being  collected  and  analyzed.... Therefore,  the  performance  of... automatic  test 
equipment  could  not  be  measured  and  ev^uated  accurately."  It  stated  that  the 
ability  to  collect  and  track  automatic  test  equipment  data  is  available  and  is  m 
use.  The  Automated  Test  Equipment  Reporting  Subsystem  module  of  the  Core 
Automated  Maintenance  System  provides  units  with  the  capability  to  show  the 
specific  relationships  between  automatic  test  system  shop  and  tester  replaceable 
units  and  aircraft  system  line  replaceable  unit  testability. 

Audit  Response.  At  the  Air  Force  Air  Combat  Command  we  specifically 
requested  performance  data  on  the  F-15  AIS  from  the  Tester  action  officer,  the 
officer  in  charge  of  the  F-15  AIS,  several  of  the  operational  noncommissioned 
officers,  and  the  noncommissioned  officer  dedicated  to  the  Automated  Test 
Equipment  Reporting  Subsystem  of  the  Core  Automated  Maintenance  System. 

The  Tester  action  officer  told  us  that  F-15  AIS  performance  data  were  not 
reliable.  The  noncommissioned  officers  agreed  that  the  data  collection  system 
was  unreliable  because  it  reported  inaccurate  data  and  was  difficult  to  use. 
Additionally,  the  METS  action  officer  in  the  F-15  System  Program  Offi^ 
indicated  that  there  was  no  ready  source  to  provide  accurate  failure  data  on  the 
METS.  Documents  from  ARINC  Research  Corporation  further  supported  the 
nonavailability  of  data  on  the  failure  rates  of  automatic  test  equipment. 

Air  Force  Comments.  Page  10,  para  3:  The  Air  Force  nonconcurred  with  our  14 

statement,  "The  1,000  hour  mean  time  between  failure  criterion  had  doubled 
over  the  criterion  cited  in  a  previous  requirements  document  approved  in  1988." 

It  stated  that  comparing  the  High  Mobility  Tester  Statement  of  Need  to  the 
Tester  operation^  requirements  document  was  unreasonable  because  the  two 
acquisition  programs  were  distinctly  different.  The  High  Mobility  Tester  was 
not  envisioned  as  an  F-15  AIS  rqilacement  as  is  the  Tester. 

Audit  Response.  The  Tester  operational  requirements  document  clearly  states 
that  the  High  Mobility  Tester  Statement  of  Need  is  the  basis  for  the 
requirement.  The  Tester  action  officer  stated  that  the  High  Mobility  Tester 
Statement  of  Ne^  was  originally  for  a  standardized  automatic  tester  for  several 
weapon  systems  including  the  F-16,  F-111,  and  A-7  aircraft.  The  action  offii^r 
explained  that  the  F-16  acquired  unique  automatic  test  equipment  and  the  F-111 
and  A-7  programs  no  longer  supported  a  requirement  for  the  acquisition  of 
standardized  automatic  test  equipment.  Consequently,  the  requirement  for  the 
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Tester  had  to  be  tailored  to  fit  revised  requirements  in  the  High  Mobility  Tester 
Statement  of  Need.  In  the  tailoring  process,  the  Air  Force  took  advantage  of 
the  possibility  that  manufacturers  could  produce  a  more  reliable  tester;  and  at 
the  urging  of  the  program  management  at  the  San  Antonio  Air  Logistics  Center, 
an  increased  mean  time  between  failure  of  1,000  hours  was  incorporated  into 
the  requirement.  The  action  officer  stated  that  he  had  no  basis  for  the 
performance  characteristics  in  the  Tester's  operational  requirements  document 
other  than  the  High  Mobility  Tester  Statement  of  Need. 

Air  Force  Comments.  Page  11,  para  2:  The  Air  Force  disagreed  with  our  14 
statement,  "We  noted  similar  discrepancies  in  the  Air  Force's  analyses 
supporting  other  performance  requirements  in  its  draft  operational  requirements 
document,  including  2  pallet  transportability,  4  man  set  up  and  operations, 

95  percent  system  availability  and  95-100  percent  reliability."  It  stated  that 
when  the  Air  Force  initially  attempted  to  procure  the  METS  it  anticipated  a 
tester  that  would  fit  on  a  single  pallet  and  be  2  man  portable.  A  prime  reason 
behind  redirecting  Ae  procurement  from  a  sole-source  attempt  was  that  vendors 
claimed  they  had  the  capability  to  fill  Air  Force  requirements  by  packaging  off- 
the-shelf  components  with  little  new  development.  The  requirements  me 
directly  in  concert  with  Air  Force  guidelines  to  enhance  each  unit's 
deployability. 

Many  of  the  specific  requirements  that  are  at  issue  resulted  from  the  Air  Force 
policy  decisions  made  under  the  Reliability  and  Maintainability  2000  program. 

The  1,000  hour  mean  time  between  failure  for  the  Tester  responded  to  direction 
from  the  Air  Force  Vice  Chief  of  St^f  stating  that,  "requirements  documents 
for  system  acquisitions  shall  specify  quantitative  R&M  [Reliability  and 
Maintainability]  levels  which  are  at  least  double  the  system  level  operational 
measures  of  reUability."  The  Reliability  and  Maintainability  2000  study  showed 
that  "Double  R/Half  M"  is  achievable  for  most  systems.  While  the  specific 
requirements  were  sometimes  difficult  to  support  analytically,  implementation 
of  this  policy  has  resulted  in  many  significant  gains  in  terms  of  increased 
capability  and  reduced  operating  and  support  costs. 

Audit  Response.  We  continue  to  believe  that  system  operators  should 
determine  and  develop  requirements  that  are  realistic,  germane,  and  relevant  to 
the  mission  success  of  the  F-15  aircraft.  As  we  stated  in  an  earlier  response, 

Air  Force  officials  were  unable  to  provide  any  rationale  for  the  performance 
characteristics  in  the  Tester  operational  requirements  document.  Therefore,  the 
performance  characteristics  could  be  considered  arbitrary  and  unjustified.  For 
example,  the  Air  Force  criterion  for  system  availability  was  99.8  percent  in  the 
High  Mobility  Tester  Statement  of  Need.  The  Air  Force  subsequently  reduced 
the  requirement  to  95  percent,  without  sufficient  mission-related  analysis  or 
rationale.  Although  we  noted  that  the  Air  Force  relaxed  technical  r^uirements 
when  competing  contractors  could  not  achieve  the  performance  criterion,  the 
Air  Force  would  not  relax  the  technical  requirements  for  the  IFTE.  In  another 
example,  the  Air  Force  established  a  performance  requirement  that  the  Tester 
had  to  isolate  an  electronic  fault  to  one  shop  replaceable  unit  95  percent  of  the 
time.  The  IFTE  requirement  was  90  percent  of  the  time.  Consequently,  the 
San  Antonio  program  management  office  disqualified  the  IFTE  on  this  technical 
basis. 
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Air  Force  Connnents,  Page  11,  para  2:  The  Air  Force  agreed  with  our  14 

statement,  "The  Air  Force  Air  Combat  Center  could  not  provide  data  analyses 
showing  how  the  values  or  parameters  were  selected,  the  cost  effect  of  the 
increased  values,  and  their  relationship  to  mission  success.”  It  stated  that  the 
values  and  parameters  evolved  from  an  October  1991  conference  of  Air  Force 
Command  representatives.  Based  on  their  experience,  the  Air  Force  officials 
combined  the  functionality  of  five  previous  individual  testers  in  determining  that 
the  Tester  required  a  large  jump  in  reliability  and  testing  parameters  over  the 
current  F-15  AIS  tester.  Contractors'  proposals,  based  on  the  latest  technology, 
gave  the  Air  Force  a  high  degree  of  confidence  that  the  1,000  hour  mean  time 
between  failure  would  be  met. 

Audit  Response.  The  process  and  analysis  in  determining  requirements  should 
be  systematic  and  documented  to  provide  internal  controls  and  to  comply  with 
defense  acquisition  management  guidance.  The  requirement  for  a  "long  jump  in 
the  reliability  and  testing  parameters"  should  be  fully  documented  and  its 
relationship  to  mission  success.  The  acquisition  of  the  Tester  has  an  estimated 
$269  million  life  cycle  cost.  Correspondingly,  the  requirements  for  this 
significant  investment  should  be  supported  with  thorough  analyses. 

Air  Force  Connnents.  Page  11,  para  3:  The  Air  Force  nonconcurred  with  the  14 

statement,  "Air  Force  had  inconsistencies  in  its  evaluations  of  cost  and  technical 

data  as  performed  by  personnel  at  the  San  Antonio  Air  Logistics  Center  and 

Aeronautical  Systems  Center,"  stating  that  reports  by  the  two  organizations 

were  not  contradictory.  The  Aeronautical  System  Center  review  compared 

CASS  and  IFTE  to  the  proposed  Tester,  identified  impacts  to  cost  and  schedule, 

and  modifications  to  CASS  or  IFTE  for  support  of  the  requirement.  The  San 

Antonio  study  was  a  more  detailed  parametric  analjjsis  of  the  Army  IFTE 

system  in  meeting  the  F-15  requirements,  and  its  findings  included  more  than 

the  availability  and  fault  detection  capability  of  IFTE.  The  Air  Force  officials 

at  die  Aeronautical  System  Center  and  San  Antonio  talked  to  different  people  at 

different  times  on  various  IFTE  capabilities  and  a  few  minor  inconsistencies 

resulted.  However,  the  two  independent  reports  came  to  the  same  conclusion, 

that  parametric  capabilities  of  the  IFTE  must  be  modified  or  upgraded  to  fulfill 

the  requirements  at  a  cost  greater  than  the  proposed  Tester. 

Audit  Response.  We  do  not  believe  that  our  statement  infers  that  the  two  Air 
Force  reports  were  contradictory.  We  continue  to  believe,  however,  that  the 
conclusions  pertaining  to  the  acceptance  of  certain  parametric  data  were 
inconsistent  in  the  two  reports. 

Air  Force  Comments.  Page  11,  para  3:  The  Air  Force  disagreed  with  our  is 

statement,  "The  Air  Force’s  disqualification  of  the  Army  test  equipment  on  a 
technical  basis  was  unjustified,"  based  on  the  data  in  the  studies  the  Air  Force 
cit^,  and  the  fact  that  the  requirements  in  the  operational  requirements 
document  have  not  changed.  Even  with  modification  for  test  requirements, 

IFTE  would  not  meet  transportability  and  portability  requirements  specified  in 
the  operation^  requirements  document. 

Audit  Response.  We  continue  to  believe  that  the  disqualification  of  the  Army 
test  equipment  was  unjustified  because  the  performance  criteria  in  the 
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operational  requirements  document  were  not  ad^uately  supported.  We  noted 
from  the  Air  Force's  comments  that  transportability  and  portability  requirements 
were  changed  to  accommodate  "off-the-shelf  technology,  but  not  for  the 
Army's  IFTE  which  is  also  "off-the-shelf." 

Air  Force  Connnents.  Page  12,  para  1:  The  Air  Force  nonconcurred  with  our  is 
statement,  "The  Air  Force  had  no  official  basis  for  establishing  Ae  technical 
criteria  for  evaluating  alternatives  to  the  tester  or  for  preparing  contract 
specifications  for  a  full  and  open  competition."  It  stated  that  the  operational 
requirements  document  was  in  the  coordination  cycle  and  an  approved 
acquisition  plan  existed.  Likewise,  the  congressional  language  and  the  U.S. 
Federal  District  Court  Stipulation  for  Dismissal  firmly  establish  an  official  basis 
for  a  full  and  open  competition  acquisition. 

Audit  Response.  The  Air  Force  did  not  have  an  approved  operational 
requirements  document  based  on  a  thorough  analysis  of  perfonnance 
characteristics,  in  accordance  with  defense  acquisition  management  guidance. 
Therefore,  we  continue  to  believe  the  Air  Force  did  not  have  a  basis  for 
establishing  the  technical  criteria  for  determining  contract  specifications  and 
evaluating  alternative  proposals.  The  congressional  language  and  court 
decisions  are  discussed  in  Other  Matters  of  Interest  in  the  report. 

Air  Force  Comments.  Page  13,  para  2:  The  Air  Force  nonconcurred  with  our  le 
statement,  "The  Air  Force  did  not  have  the  organization,  guidance,  and  data 
bases  necessary  to  evaluate  and  monitor  the  acquisition  of  automatic  test 
equipment  to  ensure  that  equipment  would  satisfy  multiple  requirements."  It 
stated  that  the  Air  Force  has  established  the  Automatic  Test  Systems  Product 
Group  Management  office  at  San  Antonio.  Although  the  concept  of  operations 
for  centralizing  the  Air  Force  automatic  test  systems  management  and 
acquisition  was  approved  and  Initial  Operational  Capability  was  declared  on 
March  31,  1993,  the  office  has  functioned  in  a  leadership  role  since  1991.  The 
Automatic  Test  System  Product  Group  Manager  is  charged  with  formulating, 
coordinating  and  executing  policy  and  guidance  that  will  achieve  Air  Force 
standardization  objectives. 

Audit  Response.  We  concluded  from  our  discussions  with  the  Director  for  the 
Automatic  Test  System  Product  Group  Management  office  that  controls  were 
not  yet  in  place  for  effectively  monitoring  the  acquisition  of  standardized 
automatic  test  equipment  in  the  Air  Force.  The  Director  stated  Aat  the 
organization,  guidance,  and  data  bases  necess^  to  evaluate  and  monitor  the 
acquisition  of  automatic  test  equipment  were  all  in  the  planning  stages. 

Air  Force  Comments.  Page  13,  para  3:  The  Air  Force  nonconcurred  with  our  5 
statement,  "Air  Force.  .  .concerns  about  congressional,  legal,  and  cost 
implications  caused  by  any  changes  in  the  acquisition  strategy.  .  .are 
unwarranted."  It  stated  that  awarding  a  sole-source  contract  for  the  Army's 
IFTE  or  the  Navy's  CASS  would  be  contrary  to  the  Competition  in  Contracting 
Act.  When  the  Air  Force  issued  a  solicitation  for  the  Tester,  numerous 
contractors  submitted  proposals  for  the  Tester.  Conceivably,  Army  and  Navy 
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could  construct  a  Justification  and  Approval  for  the  sole-source  procurement  of 
standardized  automatic  test  equipment,  but,  Air  Force  believes  there  would  be  a 
low  probability  of  success  against  the  potential  protests  and  litigation. 

Audit  Response.  We  still  believe  that  the  Air  Force's  concerns  are 
unwarranted,  as  stated  in  Other  Matters  of  Interest. 

Air  Force  Comments.  Page  13,  para  4;  The  Air  Force  nonconcurred  with  our  s 

statement,  "The  Air  Force  asserted  that  Congress  made  clear  its 
intention.  .  .that  the  tester  program  was  to  proceed  in  accordance  with  the  Air 
Force’s  competitive  acquisition  strategy.  We  believe  that  the  direction  only 
made  an  exception  for  the  Air  Force  to  provide  a  cost  benefit  analysis  to  the 
congressional  defense  committees."  The  Air  Force  quoted  from  House 
Appropriations  Committee  Report  102-627,  ".  .  .the  Committee  expats  the  Air 
Force  to  conduct  a  full,  fair  and  open  competition  for  the  acquisition  of  the 
F-15  downsized  METS."  The  Air  Force  further  quoted  from  Senate 

Appropriations  Committee  Report  102-408,  reg^ding  the  Tester,  "...the 
committee  believes  that  the  Air  Force  should  continue  its  ongoing  competitive 
development  and  procurement  for  this  capability. " 

Audit  Response.  The  comments  by  the  House  Appropriations  Committee  and 
Senate  Appropriations  Committee  were  made  in  the  process  of  determining  the 
appropriations  for  FY  1993  and  were  relevant  at  the  time.  However,  we  do  not 
believe  the  congressional  language  precludes  the  Air  Force  from  ev^uating 
other  alternative  automatic  test  equipment  for  the  F-15  to  determine  their  cost- 
effectiveness. 

Air  Force  Comments.  Page  14,  para  2:  The  Air  Force  disagreed  with  our  6 

statement,  "Through  our  discussions  with  Air  Force.  .  .competition 
advocate.  .  .we  were  unable  to  determine  any  obstacle  to  purchasing  the  Army 
test  equipment.  ..."  It  stated  that  the  Air  Force  Competition  Advocate  at  San 
Antonio  believed  that  cancellation  of  the  current  competition  and  acquiring 
Army's  IFTE  would  cause  protests  and  litigation  by  the  contractors  in  the 
competition.  Additionally,  the  Air  Force  believes  that  a  new  Justification  and 
Approval  may  be  required  to  procure  IFTE  for  Air  Force  requirements  and  a 
sole-source  justification  would  be  against  public  law,  knowing  that  other 
contractors  want  to  bid  on  the  requirement. 

The  Air  Force  also  stated  that  our  discussion  on  page  14,  para  5  was  6 

misleading.  The  Air  Force  stated  that  the  acquisition  was  for  the  METS,  which 
was  already  the  standard  automatic  test  equipment  for  the  Special  Operating 
Forces.  Originally,  the  Air  Force  Competition  Advocate  had  planned  to  direct 
a  competition,  but  approved  the  Justification  and  Approval  when  it  was 
demonstrate  that,  unless  the  Air  Force  want^  to  discard  $12  million  worth  of 
Government  equipment,  it  could  not  have  a  fair  competition. 

Audit  Response.  The  Air  Force's  comments  provide  a  general  summary  of  the 
discussion  with  the  San  Antonio  Air  Logistics  Center  competition  advocate. 

However,  in  addition  to  the  inconclusiveness  of  the  Air  Force  representative's 
opinion  during  our  audit,  he  was  unable  to  provide  any  documents  on  the 
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subject  because  he  was  not  involved  in  the  acquisition.  Therefore,  we 
considered  the  competition  advocate's  conclusions  to  be  speculation  and  not 
based  on  the  full  facts  of  the  acquisition. 

We  believe  our  statement  about  the  legal  circumstances  of  the  acquisition  of  the 
Special  Operations  Forces  automatic  test  equipment  was  accurate.  The  Gener^ 
Accounting  Office  denied  the  protest  of  Allied-Signal,  Inc. ,  because  the  Air 
Force  was  able  to  show  that  the  Government  would  lose  significant  funds  if  it 
changed  to  a  competitive  acquisition.  We  further  believe  that  the  Air  Force 
should  be  able  to  justify  buying  standardized  automatic  test  equipment  if  it  were 
to  lose  significant  funds  by  following  the  competition  strategy  of  buying  unique 
automatic  test  equipment. 

Air  Force  Comments.  Page  15,  para  3:  The  Air  Force  nonconcurred  with  our  7 
statement,  "The  Air  Force's  acquisition  of  new  automatic  test  equipment  may 
not  be  critical  or  cost  effective.  .  .Consequent,  mission  urgency  and  cost 
avoidance  do  not  appear  to  be  adequate  justincations  for  accelerating  the 
acquisition."  It  stated  that  the  audit  report's  quote  of  a  $75,000  annual  cost  for 
repairing  the  test  equipment  is  inaccurate,  and  apparently  based  on  estimates 
from  the  ARINC  study  for  repair  of  certain  stations  only.  This  does  not  equate 
to  the  actual  costs  incurred  for  AIS  repair.  The  actual  AIS  repair  costs  in 
FY  1992  were  $13.73  million  ($10.05  million  for  contract  repair  and 
$3.68  million  for  organic  repair).  It  is  anticipated  that  AIS  support  will  become 
increasingly  expensive  due  to  increasing  labor  rates  and  cost  of  repair  parts. 

The  Air  Force  also  offered  up  to  $81.9  million  of  offsets  to  help  procure  the 
Tester.  All  the  spares  and  stock  funding  offsets  ($65  million)  came  from  the 
termination  of  spares  for  the  old  AIS,  based  on  the  schedule  replacement. 
Remaining  funds  were  taken  from  the  AIS  modernization  program.  If  the 
source  selection  is  terminated,  additional  funding  will  be  required  for  station 
upgrade  and  maintenance. 

A  delay  in  the  program  would  also  impact  the  F-15  mission  supportability. 
Because  of  the  F-15's  low  avionics  reliabilify,  the  AIS  deploys  as  part  of  the 
aviation  package  and  is  required  to  be  operational  in  theater  by  day  seven.  The 
Air  Force  experienced  the  problem  of  not  being  able  to  properly  deploy  the  AIS 
during  Desert  Storm  because  of  airUft  constraints.  Conversely,  the  METS  wm 
easily  deployable,  since  it  only  required  a  single  pallet  position.  The  Tester  will 
require  only  two  pallets,  compared  to  the  14  pallets  currently  needed  for  the 
five  AIS  stations  replaced  by  the  Tester,  reducing  airlift  requirements  by  an 
entire  C-141  per  AIS. 

The  audit  report  does  not  indicate  the  basis  for  the  claim  of  up  to  $287  million 
in  estimated  expenditures  for  modernization  and  repair  parts  for  the  AIS.  This 
may  well  have  included  costs  for  technical  changes  to  the  aircraft,  which  would 
not  be  affected  by  replacing  the  AIS  with  the  Tester.  The  San  Antonio  Air 
Logistics  Center  spent  $14.2  million  not  $287  million  on  the  AIS  modernization 
program.  This  modernization  effort  was  driven  by  aircraft  line  replaceable  unit 
modification,  not  to  upgrade  AIS  reliability. 
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Audit  Response.  The  $75,000  annual  cost  for  repairing  the  test  equipment  was 
derived  from  the  ARINC  Research  Corporation  study.  The  ARINC  study  was 
the  only  documentation  provided  during  the  audit  on  the  repair  costs  of  the  F-15 
AIS  tester.  Consequently,  we  were  unable  to  determine  the  applicability  of  the 
$13.73  million  that  the  Air  Force  claimed  was  the  actual  repair  costs  for 
FY  1992.  However,  we  noted  that  the  $75,000  in  the  ARINC  study  applied  to 
three  F-15  AIS  stations  that  would  be  replaced  during  a  2-year  delay. 
Therefore,  only  the  three  stations  should  be  considered  in  the  Mnual 
maintenance  costs.  We  also  concluded  from  the  ARINC  study  that  significant 
costs  were  already  incurred  to  stock  F-15  aircraft  and  AIS  spare  parts  and  to 
upgrade  the  test  stations.  After  several  attempts  at  reconciling  the  cost  data  with 
ARINC  representatives  and  Air  Force  officials,  we  were  unable  to  determine  if 
the  funding  for  spare  and  repair  parts  and  the  modernization  was  accurate,  and 
in  which  years,  if  any,  the  program  funds  were  spent.  We  cannot  address  the 
$81.9  million  in  offsets  for  the  Tester,  which  the  Air  Force  mentioned,  because 
this  is  new  information  that  was  not  provided  during  the  audit;  however,  as 
stated  above,  we  believe  sufficient  assets  are  on  hand  to  support  the  F-15  AIS 
for  the  2-year  delay. 

To  assess  the  deployability  of  the  Air  Force's  current  and  future  test  equipment, 
the  full  complement  of  test  equipment  should  have  been  evaluated.  The  total 
complement  of  F-15  AIS  test  equipment  requires  22  pallets  for  airlifting.  If  the 
Tester  is  acquired,  the  total  complement  of  test  equipment  will  be  reduced  to 
two  pallets  for  the  Tester  and  eight  pallets  for  the  balance  of  the  F-15  AIS  test 
equipment  that  will  not  be  man  portable.  The  balance  of  test  equipment  on 
eight  pallets  will  have  to  be  transported  and  operated  in  shelters  similar  to  the 
Army's  IFTE.  Our  conclusions  are  that  Ae  Armv  IFTE  better  satisfies  the 
mission  urgency  for  deployability  because  it  could  be  modified  to  test  the  full 
complement  of  requirements  of  me  F-15  AIS  tester.  Such  modification  would 
reduce  the  total  pallets  required  to  2  1/2  pallets  compared  to  the  10  pallets 
needed  for  the  Tester  and  the  balance  of  the  F-15  AIS  tester. 

Our  $287  million  estimate  was  derived  from  the  ARINC  study,  and  discussions 
with  ARINC  representatives  and  representatives  of  the  automatic  test  equipment 
program  management  office  at  the  San  Antonio  Air  Lx)gistics  Center.  We  made 
several  attempts  at  verifying  the  costs  but  we  were  not  provided  complete  and 
accurate  information  from  Air  Force  and  ARINC  representatives. 
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Appendix  C.  Sununary  of  Potential  Benefits 

Resniting  from  Andit 


Recommendation 

Reference  Description  of  Benefit 


Amount  and/or 
Type  of  Benefit 


1. 


2.a. 


2.b. 


2.C. 


2.d. 


Internal  Control.  Nonmonetary. 

DoD-wide  policy  and  guidance  on 
standardized  automatic  test 
equipment  will  be  developed. 

Economy  and  Efficiency.  Funds  Put  to  Better  Use. 

The  Air  Force  will  avoid  purchasing  Undeterminable. 

automatic  test  equipment  that  is  not 

standardized  or  properly  justified 

with  cost,  technical,  and 

requirements  analyses. 

Economy  and  Efficiency.  Nonmonetary. 

Ensures  acquisition  of  new  automatic 
test  equipment  will  be  standardized 
and  will  be  based  on  realistic 
performance  specifications. 


Economy  and  Efficiency.  Nonmonetary. 

Ensures  adequate  cost  tmalyses  and 
comparisons  of  alternative 
standardized  automatic  test 
equipment  are  performed. 


Internal  Control.  Nonmonetary 

Air  Force  policy  and  guidance  on 
standardize  automatic  test 
equipment  will  be  developed. 
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Appendix  D.  Organizations  Visited  or  Contacted 


Office  of  the  Secretary  of  Defense 

Office  of  the  Assistant  Secretary  of  Defense  (Production  and  Logistics),  Washington  DC 


Department  of  the  Air  Force 

Office  of  the  Deputy  Assistant  Secretary  of  the  Air  Force  (Communications,  Computers, 
and  Logistics),  Washington,  DC 

Headquarters,  Air  Force  Materiel  Command,  Wright-Patterson  Air  Force  Base,  OH 
San  Antonio  Air  Logistics  Center,  San  Antonio,  TX 
Air  Force  Air  Combat  Command,  Langley  Air  Force  Base,  VA 
Aeronautical  Systems  Center,  Wright-Patterson  Air  Force  Base,  OH 


Non-Government  Organizations 

Institute  for  Defense  Analysis,  Arlington,  VA 
ARINC  Research  Corporation,  Annapolis,  MD 
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Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  (Acquisition  and  Technology) 
Assistant  to  the  Secretary  of  Defense  for  Public  Affairs 
Assistant  Secretary  of  Defense  (Production  and  Logistics) 
Comptroller  of  the  Department  of  Defense 


Department  of  the  Army 

Secretary  of  the  Army 
Inspector  General 


Department  of  the  Navy 

Auditor  General,  Naval  Audit  Service 


Department  of  the  Air  Force 

Secretary  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Acquisition) 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Air  Force  Audit  Agency 


Defense  Agencies 

Director,  Defense  Contract  Audit  Agency 

Director,  Defense  Logistics  Agency 

Director,  Defense  Logistics  Studies  Information  Exchange 

Inspector  General,  Defense  Intelligence  Agency 

Inspector  General,  National  Security  Agency 


Non-DoD  Federal  Organizations 

Office  of  Management  and  Budget 
U.S.  General  Accounting  Office 

National  Security  and  International  Affairs  Division,  Technical  Information  Center 
National  Security  and  International  Affairs  Division,  Defense  and  National  Aeronautics 
and  Space  Administration  Management  Issues 
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Non-DoD  Federal  Organizations  (cont'd) 

National  Security  and  International  Affairs  Division,  Military  Operations  and 
Capabilities  Issues 


Chairman  and  Ranking  Minority  Member  of  Each  of  the  Following  Congressional 
Committees  and  Subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Operations 

House  Subcommittee  on  Legislation  and  National  Security,  Committee  on  Government 
Operations 
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Part  rV  -  Management  Comments 


V/ 


Principal  Deputy  Assistant  Secretary  of  Defense 
(Production  and  Logistics)  Conunents 

ASSISTANT  SECRETARY  OF  DEFENSE 

WASHINGTON,  DC  20301-8000 

may  1  7  1993 

MEMORANDUM  FOR  DEPARTMENT  OF  DEFENSE  INSPECTOR  GENERAL 

SUBOECT:  Draft  Quick-Reaction  Audit  Report  on  the  Acquisition  of  the 

F-15  Downsized  Tester  (Project  NO.3LB-5014) ,  April  27,  1993 


production  and 

LOGISTICS 


I  agree  with  your  audit  findings  and  recommendations.  The 
following  comments  are  provided  to  clarify  some  specific  points. 

While  your  recommendations  suggested  that  DoD  ATS  acquisition 
policy  and  guidance  should  be  issued  by  ASD  (P&L) ,  it  is  more 
appropriate  that  this  action  be  taken  by  the  Under  Secretary  of 
Defense  (Acquisition) .  We  have  prepared  draft  policy  and  guidance 
and  are  in  the  process  of  coordinating  it  with  other  appropriate 
organizations/  including  the  Services, 

Your  draft  audit  report  states  that  "about  $167M  in  potential 
monetary  benefits  can  be  realized  from  implementing  the 
recommendations".  I  suggest  that  this  be  modified  to  reflect  that 
this  full  cost  avoidance  can  be  realized  only  if  it  is  determined 
that  no  new  test  equipment  is  needed.  However,  I  agree  that  cost 
avoidance  would  be  realized  over  time  if  the  F-15  were  to  use  a  DoD 
standard  ATS  family  in  lieu  of  new,  weapon-peculiar  ATS.  The  OSD  ATS 
study  reported/  in  the  individual  cases  examined,  that  sizable  ATS 
acquisition  savings  would  be  possible  over  the  life  of  the  weapon 
system  when  ATS  standard  families  are  used.  On  this  point,  your 
draft  audit  report  stated  that  the  OSD  ATS  study  found  "an  average 
cost  savings  of  25  to  35  percent  could  result"  through 
standardization.  Please  note  that  these  average  savings  were 
projected  for  the  case  study  systems  examined;  this  was  not  projected 
as  an  across-the-board  savings  percentage. 

Thank  you  for  your  consideration  of  these  comments .  Please  let 
us  know  if  any  further  information  or  clarification  of  our  comments 
is  needed. 

David  Berteau 
Principal  Deputy 
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Deputy  Assistant  Secretary  of  the  Air  Force 
(Acquisition)  Comments 

DEPARTMENT  OF  THE  AIR  FORCE 

WASHtNGTON  DC 


IZMilY  1993 

OFFICE  C?  'HE  ASSISTANT  SECRETARY 

MEMORANDUM  FOR  ASSISTANT  INSPECTOR  GENERAL  FOR  AUDITING 
OFFICE  OF  THE  INSPECTOR  GENERAL 
DEPARTMENT  OF  DEFENSE 

SUBJECT:  Draft  Quick-Reaction  Audit  Report  on  the  Acquisition  of  the  F-15  Downsized 
Tester  t  Project  No.  3LB'O014) 


This  is  in  reply  to  your  memorandum  requesting  the  Assistant  Secretary  of  the  Air 
Force  (Financial  Management  and  Comptroller)  provide  Air  Force  comments  on  the 
subject  report. 

The  Air  Force  supports  using  DoD  inventoried  Automatic  Test  Systems  (ATS)  and 
Equipment  (ATE)  whenever  practical  It  was  our  original  intent  to  satisfy  the  F-15 
requirement  with  a  fielded  system,  the  F-15  Mobile  Electronic  Test  Set  (METS).  Having 
since  been  directed  toward  a  competitive  acquisition  strategy,  we  have  found  that  reliable, 
cost  effective  alternatives  exist  within  the  commercial  marketplace.  Our  analyses  indicate 
that  the  proposed  Downsized  Tester  (DST)  will  be  less  expensive,  will  meet  the  program 
schedule,  and  will  be  better  suited  to  the  requirement  than  alternative  test  systems.  The 
following  discussion  of  findings  in  your  audit  report  is  provided  for  your  use  in 
reevaluating  your  conclusions  and  recommendations  tor  this  program 

Prior  Audit  Coverace 

P  2.,  Para  2.  Your  history  is  incomplete.  In  April  1992,  DoD  IG  Audit  Report  No. 
92-037,  Effectiveness  of  the  Air  Force  s  Iniemal  Controls  over  the  Development  and 
Acquisition  of  Maintenance  and  Diagnostic  Systems,  January  23,  1992.  was  referred  to 
the  Deputy  Assistant  IG  for  Audit  Followup  for  mediation  because  it  was  believed  that  the 
Air  Force’s  comments  to  the  audit  did  not  specifically  concur/nonconcur  with 
recommendation  2  In  an  April  30,  1992  memorandum,  the  DoD  IG  asked  the  Air  Force 
to  negotiate  a  mutuadv  acceptable  course  of  action  in  meeting  the  intent  of  the 
recommendation  s 

The  resultant  agreement  was  contained  in  a  DoD  IG  Memorandum  for  the  Record, 
June  30.  1992  that  an  oiganization  would  be  created  that  would  centrally  manage 
acqui.suionandci<:rbunonnf  ATE  The  Air  Force  agreed  to  promulgate  policy  that 
would  require  wO.>:  .''enefit  analysis  or  existing  systems  in  the  DoD  inventory  for  use  as 
ATE/ATS  The  Air  Force  also  agreca  mat  exemption  of  this  requirement  or  from 
implementation  ."i  results  ot  the  anas  %  sis  would  be  granted  only  upon  acceptance  ot 
written  justilicario- 
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Internal  Controls 

P.  3.,  Para  2:  "Controls  were  not  effective  to  ensure  that  ATE  being  acquired  for  the 
F- 15  aircraft  was  properly  justified  and  cost  effecti\'e.'' 

Nonconcur.  The  acquisition  is  justified  anci  is  the  most  cost  effective  solution  for  this 
requirement.  Internal  control  discrepancies  discussed  in  detail  throughout  the  report  are 
adtossed  individually, 

P.3.,  Para  2:  "About  $167  million  in  potential  monetary  benefits  can  be  realized  from 
implementing  the  recommendations." 

Nonconcur.  A  delay  of  the  program  for  two  years  could  cost  the  Air  Force  over 
$245M  in  continued  support  of  the  aging  A- 15  AIS  and  the  eventual  acquisition  of  a 
replacement  test  system  at  FY96-97  market  prices.  In  addition,  potential  lawsuits  by  DST 
bidders,  who  could  rightfully  expect  reimbursement  for  costs  incurred  in  the  bidding 
process  could  result  in  additional  monetary  losses.  Procurement  of  an  IFTE  under  the 
Army's  current  contract  would  cost  the  Air  Force  approximately  $215M,  according  to  the 
IFTE  Program  Management  Office,  as  opposed  to  the  $167M  budgeted  for  the  DST. 

Background 

P  4.,  Para  3:  "As  an  interim  measure  .(House  Appropriations  Committee)  Report 
102-1015  required  .  a  cost  benefit  analysis  comparing  newly  recommended  systems 
acquisitions  to  the  Army  (Integrated  Family  of  Test  Equipment)  IFTE  and  the  Navy 
(Consolidated  Automated  Support  System)  CASS." 

Nonconcur,  Tlie  cost  benefit  analysis  is  to  "include  a  comparison  of  the  recommended 
systems  with  existing  systems  within  the  Department  of  Defense,  the  IFTE  or  CASS 
system,  a  modified  IFTE  or  CASS  system,  and  modular  components  of  existing  systems." 
Report  102-1015  also*  slates  that  "These  restrictions  on  ATE  apply  to  all  new  weapons 
systems  being  developed  and  to  modifications  for  upgrading  existing  avionics  systems, 
except  for  the  F-15  downsize  tester  contract  which  is  to  be  awarded  in  the  near  future." 

Regarding  the  statement  on  P.5.,  Para  2,  it  should  be  noted  that  the  referenced 
Februiy  12,  1993  S  AF/AQ  memorandum  replying  to  ASD  (P&L)  provided  a  history  of 
this  acquisition  program  that  included  analyses  of  the  Army's  DFTE  and  Federal  District 
Court  litigation  over  a  sole  source  attempt  to  procure  DoD  inventoried  ATE.  Recognizing 
that  adequate  competition  does  exist,  the  Air  Force  could  not  justify  redirection  of  this 
acquisition  to  a  sole  source  contract  program,  such  as  the  Army's  IFTE. 
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Discussion 

P.6.,  Para  1:  "The  Air  Force  did  not  comply  with  defense  acquisition  management 
policies  and  procedures  for  the  acquisirion  of  the  tester...a  full  range  of  alternatives  must 
be  considered  before  deciding  to  initiate  a  new  acquisition  program." 

Concur.  Although  the  Operational  Requirements  Document  (ORD)  has  since  been 
approved,  the  Solicitation  and  Source  Selection  proceeded,  based  on  a  draft  ORD.  A 
range  of  alternative  ATS  were,  in  fact,  considered  and  the  Air  Force  originally  attempted 
to  procure  an  inventoried  system,  the  METS,  for  this  requirement.  However,  in  April 
1991  Allied-Signal  Inc.  filed  a  lawsuit  against  the  Air  Force  in  the  United  States  District 
Court  for  the  District  of  Delaware,  challenging  the  sole-source  acquisition  strategy.  The 
lawsuit  was  resolved  after  the  Air  Force  agreed  to  compete  the  acquisition  or  advise  the 
contractor  within  24  hours  if  a  Justification  and  Approval  (J&A)  tor  other  than  full  and 
open  competition  was  approved. 

P.6.,  Para  2:  "The  Air  Force  did  not  implement  recommendations  in  audit  report  No. 
92-037  to  approve  developments  of  new  ATE  only  when...it  would  not  be  cost  effective 
to  acquire  the..  Army’s  IFTE  and  Navy's  CASS...The  Air  Force  disregarded  our 
recommendation  and  in  April  1992  it  issued  a  request  for  technical  and  cost  proposals  for 
the  Tester  without  evaluating  the  Army  or  Navy  systems. ' 

Nonconcur.  Tlie  solicitation  process  had  already  begun  when  the  Report  was  issued. 
The  draft  solicitation  was  released  for  industry  comments  in  early  February  1992,  prior  to 
receipt  of  audit  Report  No.  92-037  The  Request  For  Proposals  (RFP)  was  released  in 
April  1992.  in  response  to  the  user's  requirements,  just  prior  to  the  timeframe  in  which  the 
DoD  IG  first  requested  mediation  of  the  report's  recommendations.  The  Air  Force  was 
neither  directed  nor  requested  to  hold  any  of  it’s  ATE  acquisition  programs  in  abeyance 
pending  the  mediation  of  this  report. 

P.7  ,  Para  1.  "The  final  report  on  (Air  Force  ATE  .-Xcquisition  Review)  Phase  I  did  not 
adequately  address  the  cost  and  technical  comparisons  needed  to  assess  standardization  of 
ATE...  Air  Force  cost  assessments  for  the  Tester  cannot  be  relied  upon  for  significant 
acquisition  decisions  because  the  assessments  were  incomplete  and  were  not  properly 
prepared  and  validated  ..a  cost  and  operational  effecii\  eness  analysis  (COEA)  should  be 
prepared  early  in  the  acquisition  process  ..and  approved  by  the  Air  Force  Acquisition 
Executive.. .Instead  of  a  COEA,  the  Air  Force,  initially  contracted  with  ARINC  Research 
Corporation  for  a  study  that  identified  cost  savings  that  would  be  realized  from  purchasing 
new  ATE  versus  extending  the  life  of  the  F-15  AIS." 

Nonconcur.  Ptirt  4E  of  DoDI  5000  2,  Paras  2.a-b ,  state:  "Cost  and  operational 
effectiveness  analysis  shall  be  prepared  and  considered  at  milestone  decision  reviews  of 
acquisition  category  I  programs.  The  underlying  principles  and  analytical  concepts  of  this 
section  shall  be  tailoied  and  implemented  in  support  of  acquisition  category  II,  III,  and  IV 
programs  as  deemed  appropriate  by  the  DoD  Component  .Acquisition  Executive. 
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Similarly,  AFR  57-1,  August  1992,  Para  2.b.  states:  "COEAs  are  required  for  ACATI 
(waivers  may  be  granted),  or  as  directed  by  the  applicable  Acquisition  Executive." 

DoDI  5000.2,  Air  Force  Sup  to  DoDI  5000.2,  and  AFR  57-1  establish  a  firm 
requirement  for  a  COEA  for  ACAT  I  programs  only.  In  DoDI  5000,2,  the  DAE  delegates 
management  responsibility  for  ACAT  11,  III,  and  IV  programs  to  the  Service  Acquisition 
Executive  (SAE).  The  F-15  DST  is  an  ACAT  IV  program,  and  did  not  require  a  COEA 
under  the  OSD  acquisition  guidance  contained  in  the  5000  series  DoDD/DoDIs. 

Recognizing  that  a  decision  not  to  accomplish  a  COEA  does  not  negate  the 
requirement  to  accomplish  a  cost  analysis,  it  should  be  noted  that,  prior  to  the  release  of 
the  RFP,  five  supponability/cost/capability  studies  were  performed: 

a.  SA-ALCAIMT  study  (14  February  198S):  This  study  identified  problem  elements 
within  the  F-15  AIS  at  the  time. 

b.  McAir  study  (10  February  1988):  Feasibility  and  cost  of  expanding  the  METS  to 
replace  five  AIS  test  stations  for  the  F-15  A-D  a.nd  three  for  the  F-15E,  thus  standardizing 
test  equipment  for  the  entire  F-15  fleet. 

c.  HIMOT:  A  27  March  1989  study  to  determine  if  an  existing  tester  possessed  the 
capability  to  test  specific  Line  Replaceable  Units  (LRUs)  with  minimal  modification  and 
risk.  Cost  estimates  were  determined  as  part  of  i.he  study 

d.  SA-ALC/ACC  (now  FM)  Study  (June  .989):  A  comprehensive  analysis  and  cost 
study  assessed  the  overall  economy  (including  manpower  and  training  costs)  of  using  the 
METS  to  replace  all  or  ponions  of  the  existing  F-15  AIS. 

e.  ARINC  Supportability  Study  (June  1991)  A  contractor  performed  study  analyzed 
the  supportability  of  three  AIS  test  stations. 

Collectively,  these  studies  provided  Air  Force  management  with  the  tools  necessary  to 
make  informed  decisions  and  selection  of  aliemaives.  While  the  ARINC  study  is  the  only 
one  referenced  in  your  report,  we  note  that  >  our  auditors  do  not  indicate  disagreement 
with  the  findings  of  the  studies  referenced  abo\  e  .-Xs  such,  we  assume  agreement  with  the 
cost  savings  of  approximately  $25. 13M  per }  ear  that  would  be  realized  from  replacing  the 
AIS  with  a  system  similar  to  the  METS. 

P.8.,  Para  1:  "Air  Force  officials  were  unab.e  to  provide  adequate  evidence  to  show 
that  the  Army  and  Navy  program  offices  veriiiea  the  cost  and  technical  data  used  in  the 
Air  Force's  cost  assessment." 

Nonconcur.  At  the  time  of  the  DoD  IG  \  i?::  :o  ASC,  the  cost  assessment  was  not 
complete  and  formal  Army  and  Navy  cooidina*  ?n  was  not  yet  appropriate.  ASC/ENE 
officials  explained  to  the  auditors  that  the  Arm*  :nd  Navy  hud  assigned  members  to  the 
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assessment  team  and  that  the  team  members  were  providing  Army  and  Navy  cost  figures. 
ASC  officials  understood  that  the  auditors  phoned  the  Amy  and  Navy  to  verify  their 
participation.  We  now  have  a  letter  of  coordination  on  the  final  repon  from  Mr.  Joseph 
Rivamonte,  Deputy  IFTE  Program  Manager  for  the  Army  (attached)  and  verbal 
coordination  from  Capt  Ed  Holder  (USMC),  Navy  CASS  Program  Management  Office, 
NAVAIR  55223,  23  March  1993,  whose  written  coordination  is  expected  by  28  May 
1993. 

P.8.,  Para  1:  "The  Air  Force's  actions  imply  that  the  Air  Force  had  little  or  no  intention 
of  changing  its  acquisition  strategy  regardless  of  the  outcome  of  the  cost  assessment  of  the 
tester  compared  to  the  Army  and  Navy's  test  equipment." 

Nonconcur.  The  Air  Force  was  in  Source  Selection  at  the  time  of  the  Review  and  had 
neither  been  directed  nor  requested  to  stop  the  process  The  program  was  not  even 
questioned  by  ASD  (P&L)  until  October  28,  1992  in  a  request  for  a  briefing. 

P.8.,  Para  2:  "The  Air  Force's  cost  study  and  cost  assessment  were  not  adequate 
substitutes  for  the  COEA...The  cost  assessment  was  inadequate  because  it  did  not  include 
evaluations  of  cost,  performance,  and  schedule  tradeoffs  for  differences  in  the  alternative 
Army  and  Navy  standardized  ATE,  as  required  by  DoD  guidance." 

Nonconcur.  The  cost  assessment  accomplished  by  ASC  was  not  intended  to  be  a 
substitute  for  a  COE  A,  but  was  a  comparative  cost  analysis  of  SA-ALCs  F-15  DST 
approach  versus  adapting  the  Army  and  Navy  ATE  for  the  same  purpose.  The  assessment 
prtxess  was  designed  to  provide  enough  scope  and  precision  for  management  decisions 
and  selection  of  alternatives.  The  analysis  was  part  of  a  comprehensive  Air  Force  ATE 
Acquisition  Review  that  examined  all  ATE  acquisition  programs  to  determine  the 
technical  and  programmatic  impacts  of  acquiring  IFTE  or  CASS  instead  of  the  programs' 
original  ATE  selections.  The  Review's  analysis  process  was  validated  by  the  Air  Force 
Audit  Agency  in  a  letter  dated  5  March  1993  This  endorsement  was  communicated  to  the 
auditors. 

Several  benefits  of  standardization  were  part  of  the  F-15  DST  assessment.  Economies 
of  scale,  when  Air  Force  buys  were  combined  with  Army/Navy  buys  were  included. 
Support  equipment  for  the  testers  (calibration  equipment,  self  test  software,  etc.)  was 
shared  when  multiple  testers  were  needed  at  a  single  site.  The  assessment  took  advantage 
of  savings  because  development  of  IFTE  and  CASS,  and  associated  items  (training 
courses,  software  tools,  etc.)  have  already  been  paid  for 

Other  potential  standardization  savings,  such  as  multiple  weapon  system  support  in  a 
composite  wing  and  \\orkload  efficiencies  were  not  identified  for  F- 15  DST  The  Air 
Force  does  not  presently  have  a  requirement  for  an  Intermediate  Level  composite  wing 
tester. 
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P.  9.,  Para  1:  "The  tester  that  the  Air  Force  is  competing  among  several  contractors 
will  have  projected  life  cycle  costs  of  $269  million  and  will  test  only  some  subsystems, 
electronic  modules,  and  components  from  the  F-IS  aircraft.  The  Army  and  Navy  test 
systems  can  be  modified  to  test  the  electronic  modules  from  several  avionics  subsystems 
and  several  different  aircraft  to  include  the  F-15." 

Nonconcur.  While  the  Air  Force  agrees  that  the  IFTE  and  CASS  can  be  modified  to 
test  LRUs  from  the  F-15,  it  is  also  a  fact  that  the  proposed  DST  can  be  modified  to  test 
systems  other  than  the  F-15.  Lnlike  the  man-portable  DST,  EFTE  and  CASS  would 
require  funher  modifications  to  meet  the  F-I5's  deployment  requirements.  It  should  be 
noted  that,  although  the  ASC  assessment  projects  a  life  cycle  costof  S269M  for  the 
proposed  DST,  it  also  projects  $351M  for  CASS  and  $302M  for  IFTE  for  the  F-15 
requirement,  assuming  no  slippages  to  the  current  program  schedule.  Slippage  to  the 
current  program  schedule  to  accommodate  IFTE  would  cost  an  additional  amount  of 
approximately  S13.5M  per  year  in  maintenance  costs  for  continued  suppon  of  the  F-15 
AIS. 

P.  9.,  Para  2:  "The  Air  Force’s  cost  assessment  did  not  adequately  compare  and 
evaluate  the  operating  and  support  costs  of  the  alternative  Army  and  Navy  test 
equipment...the  Air  Force  had  insufficient  cost  data  to  compare  and  evaluate  the  Tester 
with  the  Army  and  Navy’s  sysiems...Without  complete  data.  Air  Force  system  developers 
cannot  compare  and  evaluate  life  cycle  costs  for  each  alternative  ATE." 

Nonconcur.  The  O&S  cost  area  was  thoroughly  examined  by  the  ATE  Acquisition 
Review  team  and  changes  continued  to  be  made  after  the  auditors’  visit  to  ASC.  O&S 
costs  for  the  testers  and  test  program  sets  (TPS)  were  also  considered. 

Tftster  O&S  -  Both  the  Army  and  Navy  provided  tester  O&S  estimates  based  on 
projections  for  organic  repair  costs  for  their  ATE.  Since  organic  repair  for  the  testers  has 
not  started,  neither  has  actual  cost  data.  The  team  dissected  these  O&S  estimates  to  be 
sure  the  same  or  equivalent  factors  were  used  in  each  and  that  the  factors  used  were 
relevant  to  Air  Force  scenarios.  These  Army  and  Navy  estimates  were  then  used  for  all 
CASS/IFTE  assessments.  Some  Air  Force  programs,  such  as  the  F-15  DST,  did  not  yet 
have  O&S  estimates.  In  these  cases,  the  assessment  team  used  an  average  of  the  Army  and 
Navy  costs.  This  was  applied  as  a  percentage  of  ATE  production  costs  to  account  for 
differences  in  complexity  of  the  testers 

TPS  O&S  -  The  assessment  team  definition  of  TPS  includes  test  software,  adapters, 
and  tech  manuals.  There  is  \er\'  little  cost  data  available  in  the  TPS  O&S  area.  Here,  the 
team  used  an  average  of  Navy  and  Air  Force  data  in  the  1992  DoD  ATS  Investment 
Strategy  Study.  The  team  used  a  fixed  percentage  of  2  6%  of  TPS  acquisition  costs  as  the 
estimate  for  all  alternatives  The  percentage  approach  accounts  for  any  differences  in  TPS 
complexity  In  the  DST  case,  the  team  also  estimated  TPS  development  and  production 
costs  to  be  the  same  tor  all  alternatives.  It  is  our  experience  that  TPS  costs  vary  little 
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across  testers  with  similar  capabilities  This  is  the  case  in  our  F-15  DST  assessment,  where 
we  included  modifications  of  CASS  and  IFTE  to  provide  DST  equivalent  test  capabilities. 

P.  10.,  Para  1:  (technical  Assessment)  ‘The  Air  Force  performed  insufficient  analyses 
of  performance  data  and  requirements  for  new  ATE  to  support  its  requirements 
assessment." 

Nonconcur.  The  SA-ALC  Technical  Assessment,  22  December  1992,  compared  the 
F-15  LRU  performance  requirements  to  the  available  stimuli,  measurement,  and  power 
capacities  of  the  IFTE  system.  IFTE  operational  short  falls  penaining  to  Mean  Time  ^ 
Between  Failure  tMTBF),  transport  configuration,  and  portability  were  noted.  In  addition, 
fault  isolation  shorcfalls,  human  factors  limitations,  operational  deficiencies,  and  necessary 
modifications  for  the  IFTE  system  were  identified,  to  include: 

Additional  Insmiments  Required. 

a.  LRU  C(X)ling  Air  Supply 

b.  Pneumatic  Measurement  Device 

c.  Oscilloscope 

d.  Electro-Optics  /  Display  Testing  Devices 

e.  AM  /  FM  Phase  Noise  Measurement  Device 

Modifications  /  Enhancements  Required: 

a.  DC  Power  Supply  accuracy  and  current  capacity  upgrades 

b.  Frequency  /  Time  Interval  Counter  range  extension 

c  RF  Synthesizer  power  range  increase 

d.  Additional  Bussing  (H009  and  1553)  devices 

e.  Digital  pin-out  number  increases 

P.IO ,  Para  2:  "The  Air  Force  did  not  perform  sufficient  analyses  to  support  MTBF 
technical  requirements  in  the  draft  ORD" 

Nonconcur.  Experience  with  the  F-15  METS  during  System  Performance 
Demonstration  produced  an  831  hour  .VITBF  for  a  similar  device.  The  1,000  hr  MTBF 
requirement  grew  out  of  the  Air  Force  s  R&M  2000  initiative,  discussed  at  greater  length 
in  response  to  P  11,  Para  2. 

P  10,  Para  2.  ’  Officials  at  ACC  could  not  provide  any  documentation  or  other 
analyses  to  show  the  need  for  specific  performance  criterion.,  personnel  indicated  that 
such  performance  data  were  not  being  collected  and  analyzed  ..Therefore,  the 
performance  of  ...ATE  could  not  be  measured  and  evaluated  accurately." 

Nonconcur.  The  ability  to  collect  and  track  ATE  data  is  a\  aiiable  and  is  in  use.  The 
Automated  Test  Equipment  Reponing  Subsystem  ( ATERS;  irodule  of  the  Core 
Automated  Maintenance  System  (C.AMS)  provides  units  with  the  capability  to  show  the 
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specific  relationships  between  ATS  Shop/Tester  Replaceable  Units  and  aircraft  system 
LRU  testability. 

P.  10,  Para  3:  "The  1,000  hour  MTBF  criterion  had  doubled  over  the  criterion  cited  in 
previous  requirements  document  approved  in  1988  ..Officials  at  the  ACC,..could  not 
explain  why  the  MTBF  for  the  new  test  equipment  increased  from  500  hours  in  1988  to 
1,000  hours  in  1993." 

Nonconcur.  Citing  discrepancies  in  the  jump  from  500  hours  to  1,000  hrs  in  the  MTBF 
requirements  from  the  HIMOT  SON  to  the  DST  ORD  is  unreasonable,  since  the  two 
acquisition  programs  are  distinctly  different  with  separate  objectives.  The  HIMOT  effort 
sought  the  purchase  of  a  flight  line  tester  which  would  test  only  certain  "high  burner" 

LRUs  from  various  weapon  systems.  It  was  not  envisioned  as  an  AIS  replacement  as  is  the 
DST.  ACC  referenced  the  HIMOT  SON  in  the  DST  ORD  because  it  was  pertinent  to  the 
program  and  the  downsized,  highly^mobile  concept  was  still  valid.  However,  it  is 
important  to  note  that  the  two  were  distinctly  separate  programs.  The  requirement  stated 
in  the  approved  ORD  is  for  the  DST  to  "achieve  a  minimum  observed  configuration 
MTBF  of  1,000  hours  under  worst  case  environmental  conditions." 

P.  1 1.,  Para  2:  "We  noted  similar  discrepancies  in  the  Air  Force’s  analyses  supporting 
other  performance  requirements  in  its  draft  ORD,  including  2  pallet  transportability,  4  man 
set  up  and  operation,  95%  system  availability  and  95-100%  reliability." 

Nonconcur.  When  the  Air  Force  initially  attempted  to  procure  the  METS  we 
anticipated  a  tester  that  would  fit  on  a  single  pallet  and  be  two  man  ponable.  A  prime 
reason  behind  redirecting  the  procurement  from  a  soie  source  attempt  was  that  vendors 
claimed  they  had  the  capability  to  fill  Air  Force  requirements  by  packaging  off-the-shelf 
components  with  little  new  development.  As  a  result,  the  initial  draft  ORD  and  the  draft 
RFP  stated  requirements  for  a  single  pallet  and  2  man  portability.  Following  receipt  of 
industry  comments  to  the  draft  RFP,  requirements  were  modified  slightly,  since  an  "off- 
the-shelf’  solution  is  solicited.  The  requirements  are  directly  in  concen  with  Air  Force 
guidelines  to  enhance  each  unit’s  deployability 

Many  of  the  specific  requirements  that  are  at  issue  resulted  from  the  Air  Force  policy 
decisions  made  under  the  Reliability  and  Maintainability  (R&M)  2000  program,  outlined  in 
AFP  800  7.  The  1 ,000  hour  MTBF  for  the  DST  responded  to  direction  from  the  Air  Force 
Vice  Chief  of  Staff  staring  that,  "requirements  documents  for  system  acquisitions  shall 
specify  quantitative  R&M  levels  which  are  at  least  couble  the  system  level  operational 
measures  of  reliability."  The  R&M  2000  study  showed  that  "Double  R/Half  M"  is 
achievable  for  most  systems  While  the  specific  requirements  were  sometimes  difficult  to 
support  analytically,  implementation  ot  this  policy  has  resulted  in  many  significant  gains  in 
terms  of  increased  capability  and  reduced  O&S  cos:s  In  recognizing  the  limitations  to 
such  broad  approaches  to  R&M,  the  Air  Combat  Cc.mmand  (ACC)  has  restructured  its 
staff  to  belter  address  R&M  requiiements  analysis 
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P.  l  L,  Para  2:  "The  Air  Force  ACC  could  not  provide  data  analyses  showing  how  the 
values  or  parameters  were  selected,  the  cost  effect  of  the  increased  values,  and  their 
relationship  to  mission  success." 

Concur.  Representatives  from  the  using  commands,  HQ  USAF,  and  AFMC  met  in 
October  1991  to  develop  the  draft  ORD.  The  values  and  parameters  outlined  in  the  ORD 
grew  from  this  initial  conference.  Drawing  upon  dieir  knowledge  and  experience  with 
ATE,  those  involved  in  the  development  of  the  DST  requirements  defined  the  requirement 
for  a  highly  reliable  and  accurate  tester  The  DST  combines  the  functionality  of  5 
previously  individaul  testers.  This  required  a  large  jump  in  the  reliability  and  testing 
parameters  of  the  current  AIS.  Additionally,  the  DST  is  deployed  as  a  single  unit.  Failure 
during  operations  would  pose  a  high  risk  to  maintenance.  The  values  and  parameters  are 
consistent  in  the  draft  and  final  RFP  and  the  ORD  Contractors’  proposals,  based  on  the 
latest  technology,  give  us  a  high  degree  of  confidence  that  the  1,000  MTBF  will  be  met. 

P.  11.,  Para  3:  "Regarding  technical  values  and  parameters,  the  Air  Force  had 
inconsistencies  in  its  evaluations  of  cost  and  technical  data  as  performed  by  personnel  at 
the  SA-ALC  and  ASC." 

Nonconcur.  This  statement  infers  the  two  repons  are  contradictory.  Such  is  not  the 
case.  The  ASC  Review  compared  CASS  and  IFTE  to  the  proposed  DST,  identified 
impacts  to  cost  and  schedule,  and  modifications  to  CASS  or  IFTE  for  support  of  the 
requirement.  The  Army  and  Navy  provided  CASS  and  IFTE  performance  information  in  a 
sur\’ey  format  with  specific  definitions  of  the  performance  parameters.  This  information 
was  reverified  with  Army  and  Navy  program  offices  and  is  correct.  The  SA-ALC  study 
was  a  more  detailed  parametric  analysis  of  the  Army  IFTE  system  in  meeting  the  F- 15 
requirements,  and  its  findings  included  more  than  the  availability  and  fault  detection 
capability  of  IFTE.  ASC  and  SA-ALC  talked  to  different  people  at  different  times  on 
various  IFTE  capabilities,  and  a  few  minor  inconsistencies  resulted.  However,  the  two 
independent  reports  come  to  the  same  conclusions:  that  parametric  capabilities  of  the 
IFTE  must  be  modified  or  upgraded  to  tulfiil  the  requirements  at  a  cost  greater  than  the 
proposed  DST. 

P.  1 1  ,  Para  3:  "The  Air  Force's  disqualification  of  the  Army  test  equipment  on  a 
:echnical  basis  was  unjustified." 

Nonconcur,  based  on  the  data  in  the  studies  cited,  and  the  fact  that  the  requirements  in 
the  ORD  have  not  changed.  Even  with  modification  for  test  requirements,  IFTE  would  not 
meet  transportability  and  portability  requirements  specified  in  the  ORD. 

P  12.,  Para  1:  Operational  Recniiremerits  Document  Approval.  "The  Air  Force  had  no 
official  basis  for  establishing  the  technicai  criteria  for  evaluating  alternatives  to  the  tester 
or  for  preparing  contract  specifications  for  a  full  and  open  competition." 
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Nonconcur.  Official  basis  for  technical  criteria  and  contract  specifications  did  exist. 

The  ORD  was  in  the  coordination  cycle  and  an  approved  acquisition  plan  existed. 
Likewise,  House  Appropriations  Committee  (HAC)  language  in  HR  102-627,  Senate 
Appropriations  Committee  (SAC)  language  in  SR  102-408,  and  U.S.  Federal  District 
Court  Stipulation  for  Dismissal  firmly  establish  an  official  basis  for  a  full  and  open 
competition  acquisition, 

P.  12.,  Para  1:  "The  development,  processing  and  approving  of  an  ORD  is  out  of 
sequence  with  sound  acquisition  management  procedures.  The  ORD  should  have  been 
prepared  and  approved  before  RFP  was  issued  in  January  1992," 

Concur.  However,  attempts  were  made  in  good  faith  by  the  Air  Force  to  follow  proper 
acquisition  management  procedures.  The  F-15  DST  Program  Office  held  several  meetings 
with  ACC,  HQ  USAF,  AFMC,  ALC,  and  F-15  SPD  representatives  to  validate  the  user’s 
requirements  identified  within  the  ORD.  As  a  result  of  these  conferences,  an  ORD  was 
sent  out  for  coordination  on  31  October  1991  However,  a  format  change  was  mandated 
by  AFR  57-1  (I  August  1992)  and  AFP  57-2  (1  October  1992).  Caught  in  the  midst  of  a 
lengthy  coordination  process,  the  ORD  was  pulled  back  for  reformatting  and  the 
coordination  cycle  began  anew  Realizing  the  forced  format  change  was  causing  the  DST 
acquisition  to  proceed  simultaneously  with  ORD  coordination  activities,  SA-ALC 
interfaced  with  ACC  representatives  on  a  daily  basis  to  ensure  the  solicitation  being  used 
to  procure  the  DST  was  an  accurate  representation  of  the  user's  requirements  as  reflected 
in  the  ACC  prepared  ORD.  The  basic  requirements  of  the  ORD  have  not  changed  since 
release  of  the  formal  RFP.  ACC  personnel  have  functioned  as  team  members  of  the  F-15 
DST  program  office  throughout  the  process  and  have  reviewed  all  solicitation  documents 
to  ensure  the  ORD  and  solicitation  were  in  harmony.  The  ORD  received  final  approval  by 
the  Air  Force  on  1 1  May  1993 

P.  13.,  Para  2:  "As  of  March  1993  the  Air  Force  had  no  specific,  comprehensive  policy 
on  the  acquisition  of  ATE  " 

Concur.  ATS  acquisition  policy  has  not  yet  been  released.  However,  an  Air  Force 
Policy  Directive  (AFPD  63-2)  has  been  drafted,  and  is  currently  in  the  final  coordination 
cycle  for  approval  by  the  Air  Force  Chief  of  Statf  NOTE:  In  response  to  audit  Repon  92- 
037,  the  Deputy  Assistant  Secrerar>'  of  the  Air  Force  for  Acquisition  issued  interim  ATS 
acquisition  policy  to  AFMC,  in  a  memorandum,  30  July  1992. 

P.  13  ,  Para  2:  "The  Air  Force  did  not  have  the  organization,  guidance,  and  data  bases 
necessary  to  evaluate  and  monitor  the  acquisition  of  ATE  to  ensure  that  equipment  would 
satisfy  multiple  requirements.  ’ 

Nonconcur  The  Air  Force  has  established  the  ATS  Product  Group  Management 
office  at  SA-ALC  Although  the  Integrated  Weapon  S>stem  Management  (IWS.M) 
Concept  of  Operations  (CONOPsf  for  centralizing  the  Air  Force  ATS  management  / 
acquisition  was  approved  and  Initial  Operational  Capacitity  (IOC)  was  declared  on  31 
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March  1993,  the  office  has  functioned  in  an  ATS  leadership  role  since  the  IWSM  process 
began  in  199L  The  ATS  PGM  is  charged  with  formulating,  coordinating  and  executing 
policy  and  guidance  that  will  achieve  Air  Force  standardization  objectives.  Controls  are 
being  implemented  to  ensure  all  requirements  are  considered  and  meet  the  criteria  laid  out 
in  DoDD  5000.1  and  DoDI  5000  2. 

P.  13.,  Para  3:  "Air  Force... concerns  about  congressional,  legal,  and  cost  implications 
caused  by  any  changes  in  the  acquisition  strategy... are  unwarranted." 

Nonconcur.  Reversing  our  current  acquisition  strategy  of  full  and  open  competition 
and  attempting  to  award  a  sole  source  contract  would  be  contrary  to  the  Competition  In 
Contracting  Act  Statute.  In  1991  the  USAF  attempted  to  sole  source  the  F-15  METS  and 
the  United  States  District  Court  for  the  District  of  Delaware  challenged  the  sole  source. 
The  ensuing  law'suit  was  only  resolved  when  the  USAF  agreed  it  would  advise  Allied- 
Signal  if  a  Justification  and  Approval  (J&A)  for  other  than  full  and  open  competition  was 
approved.  Because  the  USAF  could  not  prove  that  only  one  source  could  satisfy  our 
needs,  a  solicitation  for  the  DST  was  issued  in  April  1992,  for  full  and  open  competition. 
As  a  result,  numerous  sources  submitted  proposals  for  the  DST.  To  reverse  this  strategy 
and  claim  that  only  the  Army's  IFTE  or  Navy's  CASS  systems  (sole  source)  can  satisfy  the 
USAF  requirements  would  be  inconsistent  with  the  competitive  facts.  Conceivably,  a  J&A 
based  on  "standardization"  could  be  constructed.  However,  the  Army  or  Navy  would  be 
required  to  synopsize  the  sole  source  requirement  which,  in  our  opinion,  would  be  subject 
to  protest  and  litigation  and  is  a  low  probability  of  success  strategy  as  there  is  little  to 
support  a  DoD  standardization  program. 

The  Acquisition  Law  Division,  S A-ALC/JAN,  reviewed  the  Allied  Signal  stipulation  of 
Dismissal  and  has  provided  a  legal  opinion  on  its  applicability.  Attorney- Advisor  30  April 
1993  opinion  is  attached. 

P.  13.,  Para  4:  "The  Air  Force  asserted  that  Congress  made  clear  its  intennon...thai  the 
tester  program  was  to  proceed  in  accordance  with  the  Air  Force  s  competitive  acquisition 
strategy.  We  believe  that  the  direction  only  made  an  exception  for  the  Air  Force  to 
provide  a  cost  benefit  analysis  to  the  congressional  defense  committees." 

Nonconcur.  H.^C  Repon  102-627  at  133,  states  "The  Air  Force  also  tried  to  award  a 
sole  source  contract  for  the  downsized  F-15  mobile  electronic  test  sets.  This  action  was 
rejected  by  procurement  officials.  It  is  the  Committee's  view  that  a  more  open  and 
competitive  process  should  have  been  used  for  the  SOF  METS  acquisition.  In  addition, 
the  Committee  expects  the  Air  Force  to  conduct  a  full,  fair  and  open  competition  for  the 
acquisition  of  the  F-15  (downsized  METS  " 

Similarly,  SAC  Repon  102-408  at  283,  .states  that,  in  regtu-d  to  the  F-15  DST, "  the 
committee  belies  es  that  'he  AirFoice  should  continue  its  ongoing  competitive 
development  and  procurement  for  this  capability  " 
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P.  14.,  Para  2:  "Through  our  discussions  with  Air  Force  ...  competition  advocate 
we  were  unable  to  determine  any  obstacle  to  purchasing  the  Army  test  equipment../' 

This  assertion  is  incorrect.  A  memo  dated  19  Feb  1993,  by  the  SA-ALC  competition 
advocate  indicates  a  different  interpretation  of  the  meeting  with  the  auditors.  It  states: 

"All  of  the  general  discussion  revolved  around  the  question,  "Is  it  against  public 
law,  in  this  case  the  Competition  in  Contracting  Act,  to  cancel  the  competitive  acquisition 
of  the  F-15  DST  and  initiate  a  Military  Interdepartmental  Purchase  Request  (MIPR)  to  the 
Army  to  buy  AF  requirements  under  their  sole  source  IFTE  contract"  The  general 
response  was  that  this  action  could  be  construed  as  circumventing  competition,  especially 
since  we  know  that  other  contractors  can  supply  the  Government  requirements. 
Cancellation  would  definitely  bring  about  litigation/protest.  Also,  the  type/extent  of 
modification  to  the  IFTE  to  meet  AF  requirements  would  also  determine  whether  or  not  a 
new  J&A  would  be  required.  We  believe  a  sole  source  justification  would  be  against 
public  law,  knowing  that  other  sources  want  to  bid  on  the  requirement." 

Similarly,  the  discussion  on  Page  14,  Para  5  is  misleadmg.  That  acquisition  was  for  the 
METS  which  was  already  the  standard  ATE  for  the  SOF.  Originally,  the  Air  Force 
Competition  Ad%ocate  had  planned  to  direct  a  competition,  but  approved  the  J&A  when 
it  was  demonstrated  that,  unless  the  Air  Force  wanted  to  discard  512M  wonh  of 
Government  equipment,  we  could  not  have  a  fair  competinon.  The  total  acquisition  cost 
for  the  METS  was  estimated  at  $20M.  with  a  cost  of  $12M  already  incurred.  The  Air 
Force  would  ha\  e  nad  to  achieve  a  savings  from  competinon  of  60%  just  to  break  even. 

P.15.,  Para  3  The  Air  Force's  acquisition  of  new  ATE  may  not  be  critical  or  cost 
effective...  Consequently,  mission  urgency  and  cost  avoidance  do  not  appear  to  be 
adequate  justifications  for  accelerating  the  acquisition 

Nonconcur.  The  audit  Report's  quote  of  $75,000  annua,  cost  for  repairing  the  test 
equipment  is  inaccurate,  and  is  apparently  based  on  estimates  from  the  ARINC  study  for 
repair  of  certain  stations  only.  This  does  not  equate  to  the  actual  costs  incurred  for  AIS 
repair.  The  actual  .AIS  repair  costs  in  FY92  were  S13.73MtS10.05M  for  contract  repair 
and  $3.68M  for  organic  repair).  It  is  anticipated  that  AIS  s-ppon  will  become  increasingly 
expensive  due  to  increasing  labor  rates  and  cost  of  repair  Tins. 

In  addition,  the  Air  Force  offered  up  to  $8 1 .9.M  in  orYsets  to  help  procure  the  DST.  All 
of  the  spares/scock  funding  offsets  ($65M)  came  from  the  lermination  of  spares  for  the  old 
AIS,  based  on  the  scheduled  replacement  Remaining  funds  were  taken  from  the  AIS 
modernization  program.  If  the  source  selection  is  terminated,  additional  funding  will  be 
required  for  station  upgrade  and  maintenance. 

A  delay  in  the  program  would  also  impact  the  F-15  rr.iiiton  supportabiliiy.  Because  of 
the  F-15’s  low  a\  ionics  reliability,  the  AIS  deploys  as  pan  ::  the  aviation  package  and  is 
lequired  to  ce  cnerational  in  theater  by  day  seven  The  ^  n  Force  experienced  the  problem 
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of  not  being  able  to  properly  deploy  the  AIS  during  Desert  Storm  because  of  airlift 
constraints.  Conversely,  the  NIETS  was  easily  deployable,  since  it  only  required  a  single 
pallet  posidon.  The  DST  will  only  require  two  pallets,  compared  to  the  14  pallets  currently 
needed  for  the  five  AIS  stations  replaced  by  the  DST,  reducing  airlift  requirements  by  an 
entire  C-141  per  AIS. 

The  audit  report  does  not  indicate  the  basis  for  the  claim  of  up  to  $287M  in  estimated 
expenditures  for  modernization  and  repair  parts  for  the  AIS.  This  may  well  have  included 
costs  for  technical  changes  to  the  aircraft  which  would  not  be  effected  by  replacing  the 
AIS  with  the  DST.  SA-ALC  spent  $14  2M  not  $287M  on  the  AIS  modernization 
program.  This  modernization  effort  was  driven  by  aircraft  LRU  modification  and  was  not 
to  upgrade  AIS  reliability. 

P.  16:  Summary 

The  .Air  Force  supports  using  DoD  inventoried  ATS  rather  than  developing  unique 
test  systems,  whenever  practical.  This  is  borne  ou:  by  our  initial  attempt  to  procure  the 
METS  for  the  F-15A-D.  We  have  found,  however,  that  there  are  reliable  and  cost 
effecrive  alternatives  within  the  commercial  markemlace  that  can  be  readily  adapted  to  our 
requirements.  Experience  with  this  program  has  srown  that  to  be  the  case.  It  should  be 
noted  that  there  is  presently  no  official  OSD  or  Joi.ni  Service  document  identifying  either 
the  criteria  or  nomenclature  of  what  constitutes  a  DoD  "standard  family  of  ATS."  There 
are  several  test  systems  within  the  Air  Force  and  DoD  inventories  that  may  be  termed 
ATS  "families."  For  example,  the  F- 1 6  AIS  suppor.s  multiple  weapon  systems  and  has 
developed  a  downsized  capability  As  stated  earlier,  inventoried  ATS  were  reviewed  but 
were  found  not  to  be  the  most  cost  effective  solutions  for  this  particular  requirement. 

Since  August  1991  our  acquisition  strategy  has  been  to  satisfy  the  requirement  through 
full  and  open  competition. 

Additionally,  we  are  concerned  by  the  late  OSD  involvement  in  this  effort.  The  Air 
Force  s  full  and  open  competition  has  engendered  a  lot  of  contractor  interest  and  allowed 
the  best  approach  for  acquiring  a  commercial-off-"' e-shelf  tester  for  our  use.  Contractors 
have  spent  a  considerable  amount  of  bid  and  proposal  money.  These  costs  may  indirectly 
be  borne  by  the  government  if  the  source  selection  s  canceled. 

In  conclusion,  the  Air  Force's  post  cold- war  rec  firemen  is  call  for  rapidly  deployed  air 
pow'er.  often  to  tu-eas  with  limited  infrastructure  a~i  support.  We  can  no  longer  rely  on 
forward  based  forces  for  support.  To  meet  these  rew  demands,  the  Air  Force  is  taking 
deliberate  steps  to  reduce  the  large  support  requirements  which  tether  our  forces  to 
vulnerable  combat  support  bases  The  F-15  is  the  r  Force’s  premier  air  superiority 
aircraft,  and  the  DST  is  the  most  effective  ATE  sc.-iion  to  sustain  its  combat  capability  in 
forward  operating  locations 

Recommendations  tor  Coirective  AcJion 
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1.  Concur  with  recommendation  1 

2.  Non  concur  with  recommendation  2.a.  Discontinuance  of  the  full  and  open 
competition  would  result  in  monetary  losses  through  prolonged  support  of  the  AIS, 
procurement  of  a  replacement  at  FY96-97  market  rates,  or,  if  sole  source  for  IFTE,  a 
higher  current  year  cost,  and  payment  of  bidders  costs  if  litigation  ensues  from  the 
canceled  source  selection.  As  addressed  in  the  discussion  above,  commercial  capability 
already  exists  for  satisfying  this  requirement. 

3.  Non  concur  with  recommendation  2.b.  The  ORD  for  this  requirement  was  approved 
by  the  Air  Force  Vice  Chief  of  Staff  on  1 1  May  1993.  The  need  for  AIS  replacement  was 
first  identified  years  ago.  Suppon  of  the  F-15  weapon  system  cannot  be  delayed 
indefinitely  while  requirements  are  continually  reevaluated.  Rationale  for  nonconcurrence 
is  elaborated  in  the  discussion  above 


4.  Non  concur  with  recommendation  2  c.  As  stated  above,  a  COEA  is  required  for 
ACAT I  programs  only.  Authority  for  the  designation  of  this  requirement  for  ACAT II, 

III,  and  IV  programs  is  delegated  to  the  SAE.  The  Air  Force  has  decided  that  it  w'ould  not 
be  cost  effective  to  accomplish  a  full  COEA  for  this  program.  Several  Cost  analyses  have 
already  been  accomplished  for  this  acquisition 

5.  Concur  with  recommendation  2  d  The  Air  Force  intends  to  issue  policy  for  the 
acquisition  of  ATS/ATE  by  30  June  1993 


OARLEENA  DRUYUN 
Deputy  Assistant  Secretary 
(Acquisition) 


2  Attachments 

-  AMCPM-TMDE  Letter,  Air  Force  .ATE  Acquisition  Review,  12  May  1993 

-  SA-ALC/JAN  Letter,  Applicability  of  Stipulation  of  Dismissal,  30  April  1993 
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FROM:  SA-ALC/TAN  50  April  1993 

SIZSIRCT.  AppiiccWlin'  in  Tne.  v.  Urited 

States.  NcrJl-stWtJlI,  (D.  Dd.) 


TO:  U:a-2 


1.  You  cave  requested  our  opinion  on  tks  effect  of  tie  snculation  of  dismissal  signed  in 
Me7 1991  on  a  potenrial  decision  to  acquire  the  Airy’s  Iragraed  Famiiy  of  Test 
Equip--"  (3F7S)  in  lieu  of  the  Downsized  Tester. 

2,  The  scnulationrequires  us  to  notify  AJlied-Signai’s  counsel  withia  one  work  day  of 

your'beinr  notified  that  a  jusrificsrioa  and  Mprovai  fer  soi«ompetitive  contracting  hsd 
been  aooreved.  noncompetitive  negotiations  had  cessared,  or  a  noncompetitive  contract 
had  been  swarded.  We  are  not  prevented  by  the  stipaiation  talang  any  specific 

acacss. 


3.  You  have  advised  that  an  IFTE  acquisition  would  be  nos^^oritive  in  nature.  We 

tberetbre  co  not  believe  that  we  couid  achieve  a  cenraet  &r  IFTE  without  taJang  sa  sa 
that  wouic  mquire  us  to  notify  Allied-Signal,  they  would  then  be  able  to  decide  whether 
tiiey  wishes^  taimlesakacden. 


— 9 

.DAVISON  / 
Attoisey-Advisor 
Aequisitisn  Law  Dillon 


ATTOJWSy'S  WORKMODUCr  ?312?XS23  Ct  CIRTCT 
ORXNDt3ieCTA*VnC37JLnO.-f  C7  :JTT3ATIO?e. 

xorroi  rzzjsaax  dr  tzumsttsl  c- .  ottsz 

AJS  rogQl.  *An2Qt.TT!I2  STSuJ~>-  AJ?R0VAL 
oFT2SORicraTORORs:GazSA;.T2CRmr-  vor 
suajECTTC  r:::co*Trvr 

?V2  L  —  •  *  *'* 
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OEPARTMeNT  OF  THE  ARMY 

united  STa  iES  army  test,  MEASUBSMSNT.  ANO  OlAQNOETtC  EQUIPMENT  ACTIVITY 
REOSTONC  ARSENAU  ALABAMA  35866*5400 


^CFUVTO 
ATTCNTION  09 


1  2  MAY  1993 


AMCPJI-TMDB  (70-ir) 


MEMORANDIJM  for  Commanding  Officer,  wright  Patrerson  Air 

Force  Base,  ATTN:  apc/ENEM  (Craig  Wail), 
Wright  Patterson  Air  Force  Base,  Ohio 

SUBJECT:  Air  Force  (AF)  Automatic  Test  Equipment  (ATE) 
Acquisition  Review,  Wright-Patterson  Air  Force  Base,  Ohio, 
23-24  Nar  93 


t ,  References : 

a.  Air  Force  ATE  Acquisition  Review  Final  Report^ 
draft,  25  Mar  93. 

b.  Revised  Cost  Comparison  tables  from  ASC/ENECA,  dared 
2  Apr  93. 

2 .  The  draft  final  report  as  modified  by  the  revised  cost 
comparison  tables  has  been  reviewed. 

3.  The  IFTE  data  as  defined  in  the  report  is  considered  an 
accurate  characterization  of  the  Army's  ability  to^^upport 
potential  AF  programs. 


JOSEPH 'M.  RIVAIiONTE 
Deputy  Program  Manager 


AN  EQUAL  opportunity  employer 
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Shelton  R.  Young  Director,  Logistics  Support  Directorate 

Christian  Hendricks  Program  Director 

Tilghman  A.  Schraden  Project  Manager 
Hassan  A.  Soliman  Auditor 
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